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I.  INTRODUCTION 


The  United  States  Marine  Corps’  Education  Command  (EDCOM),  also  known  as 
Marine  Corps  University  (MCU),  is  tasked  with  the  development,  delivery,  and 
evaluation  of  the  professional  military  education  (PME)  conducted  through  resident  and 
non-resident  courses  throughout  the  Marine  Corps  (“MCU  Mission  Statement,”  n.d.). 
One  of  MCU’s  goals  to  continuously  improve  their  quality  of  education  is  to  “Leverage 
and  integrate  state-of-the-art  information  and  education  technologies...  to  provide 
students  a  relevant  educational  experience  in  the  most  effective  and  efficient  manner.” 
(“Goal  3,”  n.d.).  An  important  part  of  maintaining  the  readiness  of  the  Marines  under 
high  operational  tempos  throughout  the  world  is  providing  effective  and  efficient  PME  in 
the  form  of  quality  e-learning  products.  MCU  relies  on  their  subordinate  unit,  College  of 
Distance  Education  and  Training  (CDET),  to  provide  the  e-leaming  infrastructure  and 
products  needed  to  support  distance  learning. 

One  of  the  systems  CDET  uses  to  administer  e-leaming  courses  is  MarineNet. 
MarineNet  is  a  learning  management  system  (LMS)  that  allows  CDET  to  deliver 
Shareable  Content  Object  Reference  Model  (SCORM)  compliant  e-leaming  courses  and 
track  students’  completion  status.  The  SCORM  standard  was  developed  in  2000  and  last 
updated  in  2009.  Since  then,  efforts  by  experts  have  focused  on  developing  new 
standards  that  provide  a  wider  range  of  flexibility  in  the  capture,  storage,  and  analysis  of 
all  learning  experiences  that  occur  in  e-learning.  One  such  standard  developed  by 
Advanced  Distributed  Learning  (ADL)  was  Experience  API  (xAPI).  The  draft  version  of 
the  MarineNet  Five  Year  Strategic  Plan  (FY14-FY18)  lists  the  development  of  xAPI 
courseware  as  a  means  of  enhancing  their  capabilities  to  deliver  effective,  responsive,  and 
relevant  e-learning  resources  (Michael  Gavin,  personal  communication,  February  2, 
2016).  For  the  use  of  xAPI  to  be  effective  in  e-leaming,  a  thorough  analysis  needs  to  be 
conducted  of  the  types  learning  experiences  to  be  captured  in  order  to  provide  educators 
the  relevant  data  necessary  to  make  effective  decisions  pertaining  to  students,  courses, 
and  curriculums. 
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This  thesis  researches  topics  in  education,  course  structure  and  development,  and 
enabling  technologies  that  support  e-leaming  courses,  to  determine  the  learning 
experiences  to  capture  with  xAPI.  This  provides  CDET  the  types  of  relevant  data  needed 
to  improve  their  e-learning  courses,  methods  at  which  to  capture  and  store  the 
experiences  as  data  points,  and  ways  to  analyze  the  data  to  provide  a  useful  metric  to 
make  educational  decisions. 

The  capture  of  key  learning  attributes  and  the  follow-on  analysis  of  that  data 
requires  examining  a  student’s  learning  activities  to  determine  the  effectiveness  of  the 
instruction.  This  thesis  focuses  on  the  capture  of  key  learning  attributes  by  seeking  to  use 
online  assessments  to  place  a  percentage  value  on  how  much  a  student  has  learned  by 
tracking  elements  of  their  learning  activities  and  extracting  them  to  a  format  that  can  be 
used  for  follow-on  analysis  by  educators.  Specifically,  this  thesis  examines  how  learning 
objectives,  online  assessments,  and  xAPI  measure  a  student’s  performance  in  specific 
subject  areas  by  capturing  and  extracting  learning  experiences  to  a  Learning  Record  Store 
(LRS). 

A.  THESIS  OBJECTIVE 

The  objective  of  this  thesis  is  to  provide  CDET  with  a  method  of  incorporating 
xAPI  into  their  e-learning  courseware  enabling  them  to  improve  the  level  of  education. 
The  method  recommended  to  CDET  captures  and  stores  learning  experiences  using  xAPI 
that  allow  the  establishment  of  individual  and  collective  learning  objective  performance 
values.  The  calculated  performance  values  can  then  be  used  to  improve  the  level  of 
education  they  provide  by  allowing  a  more  detailed  analysis  of  a  student’s  knowledge 
leading  to  a  more  tailored  educational  experience  or  improvement  to  the  overall 
curriculum.  The  prototype  system  used  to  demonstrate  this  method  consists  of  an  e- 
leaming  course  with  xAPI  and  utilizes  existing  software  as  a  service  (SaaS)  learning 
technologies  to  create  a  new  way  to  use  learning  activities  to  support  e-learning. 
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B.  THESIS  OUTLINE 


Chapter  II  provides  research  and  background  information  that  will  assist  in 
understanding  the  prototype  system’s  purpose,  creation,  technologies,  and  end-state.  The 
following  are  the  topics  discussed:  assessment  of  student  learning,  CDET’s  current  e- 
learning  system  elements  and  architecture,  enabling  e-leaming  technologies,  and  xAPI. 

Chapter  III  discusses  the  stakeholders  and  three  use  cases  used  in  the  development 
the  prototype  system.  The  stakeholders  used  include  the  student,  course  instructor,  course 
developer,  and  institutional  administrator.  The  three  use  cases  describe  the  prototype 
courseware  creation,  capture  and  storage  of  assessments,  and  extraction  of  data. 

Chapter  IV  provides  a  full  description  of  the  prototype  system’s  architecture, 
courseware,  and  components.  Also  included  is  a  complete  demonstration  of  the  prototype 
system’s  capture  and  extraction  of  learning  experience  data,  and  follow-on  analysis  of 
student  learning  objective  performance  values. 

Finally,  Chapter  V  summarizes  the  research  conducted  and  recommends  topics 
for  follow-on  research  in  areas  related  to  assessments  of  student  learning  and  learning 
experience  data  analysis. 
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II.  BACKGROUND 


E-leaming  uses  varying  levels  of  electronic  technologies  to  access  educational 
materials.  The  amount  and  level  of  electronic  technologies  will  differ  depending  on  the 
type  of  course.  According  to  John  Sener,  e-learning  is  typically  used  in  the  following 
types  of  courses:  synchronous  distributed  courses,  web-enhanced  courses,  blended  or 
hybrid  classroom  courses,  online  courses  where  all  activities  are  conducted  online 
without  face-to-face  interaction,  or  flexible  mode  courses  (2015).  Without  face-to-face 
interactions,  it  can  be  challenging  to  determine  how  much  a  student  has  learned  during  a 
course. 

A.  ASSESSMENT  OF  STUDENT  LEARNING 

When  determining  how  much  online  course  students  have  learned,  an  important 
measure  to  evaluate  is  their  performance.  It  is  expected,  throughout  the  course  that  the 
students’  performance  in  subject-related  tasks  will  improve.  According  to  Patrick, 
Edwards,  Wicks,  and  Watson’s  report  created  for  the  International  Association  for  K-12 
Online  Learning,  two  measures  used  to  determine  changes  in  performance  are  proficiency 
and  growth  (2012).  This  same  report  states  proficiency  is  used  to  assess  students’ 
knowledge  at  a  specific  time  in  their  learning,  while  growth  assesses  students’ 
“proficiency,  skills,  and  knowledge  gained  in  a  given  period  of  time”  (Patrick  et  al., 
2012,  p.  9). 

Effective  assessments  need  to  be  administered  and  evaluated  to  gauge  student 
performance  levels  in  different  subject  areas.  The  Eberly  Center  at  Carnegie  Mellon 
University  states  “assessments  should  reveal  how  well  students  have  learned”  and 
“assessments,  learning  objectives,  and  instructional  strategies  need  to  be  closely  aligned 
so  that  they  reinforce  one  another”  (“Why  should  assessments,”  n.d.).  Course  learning 
objectives  should  be  clearly  stated  and  reflect  what  a  student  should  learn  during  the 
instruction.  Linking  assessment  questions  directly  to  individual  course  learning  objectives 
will  allow  a  student’s  proficiency  in  those  learning  objectives  to  be  determined.  By 
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determining  these  individual  learning  objective  proficiencies  a  more  detailed  assessment 
can  be  made  as  to  the  amount  of  learning  that  occurred  in  the  course. 

Comparing  students’  individual  learning  objective  proficiencies  over  time  will 
assist  in  determining  their  growth  in  the  associated  subjects.  Repeated  and  similar 
learning  objective  proficiencies  of  courses  within  a  given  subject  can  be  compared  and 
analyzed  to  assess  total  learning  growth  in  that  subject  or  their  subsets.  Figure  1  is  an 
example  of  two  different  courses  with  the  same  subject  matter  (vehicle  maintenance)  with 
overlapping  learning  objectives. 


Figure  1.  Example  Subject  and  Subset  Courses  with  Repeated 

Learning  Objectives. 


YTHICLE  MAINTENANCE  COURSES 


PASSENGER 

VEHICLES 


ObjC 


COMMERCL4L 

VEHICLES 


Obj  A  -  Know  the  components  of  a  hydraulic  brake  system 
Obj  B  -  Know  the  components  of  an  electrical  system 
Obj  C  -  Know  the  components  of  a  power  steering  system 
Obj  D  -  Know  the  components  of  an  air  brake  system 


VEHICLE  MAINTENANCE  COURSES  is  the  subject  and  PASSENGER  VEHICLES 
and  COMMERCIAL  VEHICLES  are  its  subset  courses.  Both  passenger  and  commercial 
vehicles  could  have  the  same  electrical  and  power  steering  systems  (Obj  B  &  Obj  C), 
which  could  lead  to  repeated  learning  objectives.  Typically  they  would  have  different 
brake  systems,  as  seen  with  Obj  A  &  Obj  D. 
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The  use  of  learning  objectives  proficiencies  to  determine  a  student’s  knowledge 
growth  in  a  particular  subject  area  or  one  of  its  subsets  can  be  a  method  in  determining 
their  strengths  and  weakness  in  those  areas.  Leaders  and  educators  can  analyze  this  data 
to  create  a  more  tailored  and  effective  learning  experience  for  the  student  while  providing 
a  more  efficient  use  of  educational  resources. 

B.  COLLEGE  OF  DISTANCE  EDUCATION  AND  TRAINING 

CDET  is  tasked  by  their  higher  headquarters,  EDCOM  to  “design,  develop, 
deliver,  evaluate,  manage,  and  resource  distance  learning  products  and  programs  across 
the  Marine  Corps  training  and  education  continuum  in  order  to  increase  operational 
readiness”  (“Welcome  to  CDET!,”  n.d.).  The  distance  learning  network  CDET  uses 
MarineNet  to  deliver,  evaluate,  and  manage  their  e-learning  courses.  Department  of 
Defense  Instruction  1322.26  paragraph  6.2  states  “All  acquired  or  developed  DOD 
systems  shall  conform  to  the  SCORM  (current  version)  to  ensure  accessibility,  durability, 
reusability,  maintainability,  and  interoperability”  (Department  of  Defense,  2006,  p.  4). 
CDET  conforms  to  this  requirement  by  publishing  all  their  e-learning  courseware  on 
MarineNet  as  SCORM  compliant. 

The  network  architecture  MarineNet  uses  to  support  their  e-leaming  courses,  seen 
in  Figure  2,  is  located  at  CDET’s  Distance  Learning  Operations  Center  and  consists  of  a 
LMS,  SCORM  Engine,  and  Internet  Content  Delivery  Engines  (ICDE)  and  is  seen  in 
Figure  2  (College  of  Distance  Education  and  Training,  2012).  The  LMS  is  server-based 
software  managing  the  course  catalog,  student  enrollment,  and  tracks  data  specified 
within  the  SCORM  compliant  course.  According  to  Michael  Gavin  (personal 
communication,  November  12,  2015),  CDET  currently  tracks  the  following  student  data 
in  the  LMS:  enrollment  date  and  status,  completion  date  and  status,  course  grade,  number 
of  course  launch  attempts,  total  time  course  opened,  and  varying  course  specific  custom 
data. 
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Figure  2.  MarineNet  Network  Architecture.  Adapted  from  College  of 
Distance  Education  and  Training  (2012). 


Distance  Learning  Operations  Center 


INTERNET 


HTTP 


The  SCORM  Engine  is  part  of  the  LMS  and  it  has  two  primary  functions.  First,  it 
is  used  to  make  SCORM  Application  Programming  Interface  (API)  calls  from  the  LMS 
to  the  SCORM  courseware  servers  to  display  content.  Second,  it  relays  course  and 
student  data  transactions  back  to  the  LMS  for  tracking  and  storage.  A  limitation  with 
CDET’s  LMS/SCORM  architecture  is  the  requirement  for  a  student  to  maintain  a 
constant  connection  (via  the  Internet  and  ICDE)  to  the  LMS  for  the  content  delivery  and 
tracking  to  occur. 

Most  e-leaming  courses  hosted  on  MarineNet  consist  of  self-paced  content 
modules  and  have  associated  assessments  in  the  forms  of  in-course  quizzes  or  end-of- 
course  (EOC)  exams.  Of  the  courses  that  currently  require  students  to  participate  in  an 
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assessment,  all  have  questions  that  require  only  multiple  choice  answers  (Michael  Gavin, 
personal  communication,  February  2,  2016).  In  the  MarineNet  Courseware  Development 
Technical  Standards  Version  2  document,  CDET  discourages  course  developers  from 
using  matching  text  responses  and  completely  restricts  the  use  of  essay,  spoken  voice,  or 
file  uploads  responses  in  their  assessments  (College  of  Distance  Education  and  Training, 
2012).  This  is  due  to  evaluator  requirements  to  formally  grade  performance  of  responses 
for  these  methods. 

C.  ENABLING  TECHNOLOGIES 

1.  Software  as  a  Service 

SaaS  is  a  service  model  of  cloud  computing  providing  customers  the  ability  to 
access  software  applications  on  a  service  provider’s  cloud  infrastructure  over  a  network 
or  through  the  Internet  (Grance  &  Mell,  2011).  SaaS  is  limited  in  scope  to  the  services  the 
software  provides  and  does  not  allow  a  customer  the  ability  to  influence  the  physical 
storage,  processing,  operating  systems,  location,  or  other  resources  that  Platforms  as  a 
Service  or  Infrastructure  as  a  Service  might  offer.  There  are  many  service  providers  who 
are  using  SaaS  to  provide  software  solutions  that  support  e-learning.  Three  services  this 
thesis  will  focus  on  are  course  generation  software,  LMS,  and  LRS. 

SaaS  brings  with  it  many  benefits  to  the  customer.  Costs  are  brought  down  by 
eliminating  the  requirement  to  maintain  costly  infrastructure  and  software  on  site.  The 
costly  management  of  resources  (servers,  software,  facilities,  power,  data  backups)  is 
transferred  from  the  customer  to  the  service  provider  responsible  for  maintaining  the 
infrastructure.  This  affords  smaller  customers  access  to  technologies  and  infrastructure 
that  might  otherwise  be  too  costly  to  setup  and  maintain  on  site.  It  also  gives  customers 
access  to  hardware  and  software  upgrades  without  risk  of  dismption  to  their  operations. 
Services  are  typically  available  through  most  devices  and  from  locations  with 
connections  to  the  internet.  Different  SaaS  pricing  strategies  (see  Table  1)  allow 
customers  to  pay  for  the  amount  and  type  of  services  they  require  leading  to  a  more 
efficient  use  of  money  and  resources. 
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Table  1.  SaaS  Pricing  Strategies.  Adapted  from  Jaatmaa  (2009). 


Cost  Based 

Flat  Pricing 

•  Fixed  price  for  unlimited  use  of  service,  typically 
without  up-front  fees 

Tiered  Pricing 

•  Pricing  is  based  on  package  of  services 

Performance-based  Pricing 

•  Pricing  based  on  theoretical  throughput  of  the 
system  such  as  MIPS  (Million  Instructions  per 
Second) 

User-based  Pricing 

•  Pricing  is  based  on  the  number  of  users  that 
utilize  a  collection  of  service  capabilities  over  a 
given  period  of  time. 

Usage-based  Pricing 

•  Pricing  is  based  on  customers’  actual  usage  on  a 
transaction  basis. 

Value-based 

Penetration  Pricing 

•  Market  segments  where  buyers  have  high  price 
sensitivity  are  targeted. 

Skim  Pricing 

•  Market  segments  where  buyers  are  relatively 
insensitive  to  price  and  have  high  search  costs  are 
targeted. 

Hybrid  Pricing 

•  Combines  elements  from  penetration  and  skim¬ 
pricing. 

a.  Course  Generation  Software 

Course  generation  software  allowing  users  to  author  online  courses  that  are 
SCORM  1.2/2004  compliant  and  compatible  with  a  LMS  have  become  readily  available 
as  a  SaaS.  Some  of  the  companies  that  offer  these  services  are  Easygenerator, 
SmartBuilder,  Izzui,  and  myUdutu.  These  companies  provide  varying  levels  of  services 
and  options  for  free  or  on  a  tiered  based  pricing  strategy.  Examples  tiered  pricing  is  based 
on  are  the  amount  of  courses  created,  number  of  users,  types  of  assessment  questions  and 
content  allowed  for  upload,  and  types  of  course  templates  available.  The  following  are 
the  services  a  provider’s  software  can  include:  learning  objective  creation,  basic  and 
advanced  content  upload  and  integration,  assessment  creation,  xAPI  statement  creation 
and  extraction,  online  course  hosting  with  internal  LMS  or  as  webpage,  and  publish 
courses  in  SCORM  compliant  format.  Course  generation  software  allows  a  person  with 
little  knowledge  in  code  writing  to  create  an  online  course  that  incorporates  common  e- 
learning  technologies  through  easy  to  use  graphical  user  interfaces. 
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b.  Learning  Management  Systems 

A  LMS  is  software  that  allows  the  management  and  administration  of  online 
courses  and  their  users  (students,  instructors,  administrators).  Some  companies  that  offer 
LMSs  as  SaaS  are  TalentLMS,  Litmos  LMS,  WizIQ  LMS,  and  LatitudeLeaming  LMS. 
LMS  providers  implement  pricing  strategies  based  on  the  levels  of  services  and  options 
provided  and  are  tiered  based,  usage-based,  or  user-based.  Examples  of  items  the  LMS 
pricing  strategies  can  be  based  on  are  the  amount  of  courses  and  users,  reports  and 
analytics  available,  pre-authored  course  catalog,  and  languages  available.  The  following 
are  the  types  of  services  a  provider’s  LMS  software  can  include:  course  hosting, 
assignment  of  user  roles,  curriculum  course  management,  enrollment  request 
management,  report  course  completion  status,  report  course  assessment  performance 
value,  and  administer  course  surveys.  These  LMS  services  enable  e-learning  by  providing 
a  means  for  students,  instructors,  and  administrators  to  participate  in  educational 
experiences  without  the  costly  infrastructure  typically  required  to  host  and  administer 
online  courses. 


c.  Learning  Record  Stores 

Learning  record  stores  are  data  repositories  that  store  experience  statements  sent 
from  systems  that  have  incorporated  xAPI  into  their  software.  Two  SaaS  LRSs  found 
available  were  Watershed  LRS  and  Saltbox’s  Wax  LRS.  It  is  more  common  to  find  an 
LRS  as  an  open  source  download  for  hosting  on  a  local  server.  Most  SaaS  LMS  providers 
offer  an  integrated  analytic  reporting  capability  with  varying  levels  of  user  tracking. 
Saltbox  offers  free  services  or  a  tiered  based  pricing  strategy  for  a  larger  statement 
storage  capacity.  The  following  are  the  types  of  services  Saltbox  and  other  downloadable 
LRSs  offer:  xAPI  statement  storage,  export  captured  JavaScript  Object  Notation  (JSON) 
to  comma-separated  value  (CSV),  and  statement  analytics  and  reports  based  on  activity 
and  user.  These  LRS  services  enable  e-leaming  by  providing  a  repository  for 
administrators  and  instructors  to  store  captured  student  experiences  for  later  use  in  the 
analysis  and  improvement  of  the  overall  educational  experience. 
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2.  Experience  API 

Experience  API’s  development  was  facilitated  by  ADL  Initiative  as  a  means  to 
overcome  the  limitations  in  using  SCORM  and  also  to  capitalize  on  the  new  devices  and 
technologies  capable  of  expanding  the  e-learning  environment  (“xAPI  Background  & 
History,”  n.d.).  A  focus  of  SCORM  development  in  2000  was  to  standardize  online 
course  content  to  ensure  compatibility  and  content  delivery  across  different  LMS 
platforms.  Since  2000,  the  methods  of  delivering  content  in  e-learning  has  evolved  from 
an  LMS  only  environment  to  include  reading  news  articles,  watching  online 
presentations,  using  mobile  apps,  or  simulator  training  (“Experience  API,”  n.d.).  ADL 
developed  xAPI  to  capture  learning  occurring  in  these  emerging  delivery  methods  and  for 
data  collection  to  provide  “all  types  of  learning  activity  data  that  can  be  analyzed  and 
correlated  to  productivity  and  performance  metrics”  (“xAPI  Background  &  History,” 
n.d.). 

Experience  API  was  created  as  a  collection  of  four  sub-APIs.  For  the  purpose  of 
this  thesis,  only  Statement  API  will  be  discussed.  The  Statement  API  is  used  to  track  and 
store  learning  activities  in  a  structured  data  format  (“xAPI  Architecture  Overview,”  n.d.). 
The  repository  where  all  xAPI  generated  learning  experience  statements  get  stored  is 
called  the  LRS.  ADL’s  website  states  the  LRS  is  responsible  for  storing  and  controlling 
access  to  the  data  created  by  the  statements,  and  for  the  validation  of  the  statement’s 
format  when  received  in  the  LRS  to  reduce  issues  with  other  systems  accessing  the  data 
(“ADL  LRS,”  n.d.). 

The  data  format  used  to  create  and  store  to  the  LRS  is  JSON.  JSON  was  designed 
to  be  a  data-interchangeable  language  easily  read  by  both  humans  and  machines 
(“Introducing  JSON,”  n.d.).  This  format  allows  maximum  flexibility  for  people  and 
systems  conducting  analysis  on  the  data.  The  xAPI  statement  data  model  shown  in 
Figure  3  depicts  the  eleven  top-level  properties  available  to  be  created  with  the  Statement 
API. 
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Figure  3.  Top  Level  xAPI  Statement  Data  Model.  Adapted  from  (“xAPI 

Statement  Data  Model,”  n.d.). 


The  xAPI  statement  specifications  list  the  following  three  requirements:  a 
property  cannot  be  used  more  than  once  per  statement,  each  statement  must  include  an 
actor,  verb,  and  object  property,  and  properties  may  appear  in  any  order  within  the 
statement  (“Advanced  Distributed  Learning  Co-Laboratories,”  n.d.,  Statement  properties 
section).  Table  2  includes  each  xAPI  statement  property  with  its  associated  type  of  data, 
description,  and  statement  inclusion  requirements. 
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Table  2.  xAPI  Statement  Properties.  Adapted  from  (“Advanced  Distributed 
Learning  Co-Laboratories,”  n.d.,  Statement  properties  section). 


Property 

Type 

Description 

Required 

id 

Universally 

Unique 

Identifier 

(UUID) 

UUID  assigned  by  LRS  if  not  set  by  the 
Activity  Provider. 

Recommended 

actor 

Object 

Who  the  Statement  is  about,  as  an  Agent  or 
Group  Object.  Represents  the  "I"  in  "I  Did 
This". 

Required 

verb 

Object 

Action  of  the  Learner  or  Team  Object. 
Represents  the  "Did"  in  "I  Did  This". 

Required 

object 

Object 

Activity,  Agent,  or  another  Statement  that  is 
the  Object  of  the  Statement.  Represents  the 
"This"  in  "I  Did  This".  Note  that  Objects 
which  are  provided  as  a  value  for  this  field 
should  include  an  "objectType"  field.  If  not 
specified,  the  Object  is  assumed  to  be  an 
Activity. 

Required 

result 

Object 

Result  Object,  further  details  representing  a 
measured  outcome  relevant  to  the  specified 
Verb. 

Optional 

context 

Object 

Context  that  gives  the  Statement  more 
meaning.  Examples:  a  team  the  Actor  is 
working  with,  altitude  at  which  a  scenario 
was  attempted  in  a  flight  simulator. 

Optional 

timestamp 

Date/Time 

Timestamp  (Formatted  according  to  ISO 

8601)  of  when  the  events  described  within 
this  Statement  occurred.  If  not  provided,  LRS 
should  set  this  to  the  value  of  "stored"  time. 

Optional 

stored 

Date/Time 

Timestamp  (Formatted  according  to  ISO 

8601)  of  when  this  Statement  was  recorded. 

Set  by  LRS. 

Set  by  LRS 

authority 

Object 

Agent  who  is  asserting  this  Statement  is  true. 
Verified  by  the  LRS  based  on  authentication, 
and  set  by  LRS  if  left  blank. 

Optional 

version 

Version 

The  Statement’s  associated  xAPI  version, 
formatted  according  to  Semantic  Versioning 
1.0.0. 

Not  Recommended 

attachments 

Array  of 

attachment 

Objects 

Headers  for  attachments  to  the  Statement 

Optional 

Aside  from  (potential  or  required)  assignments  of  properties  during  LRS  processing  (“id,”  “authority,” 
“stored,”  “timestamp,”  “version”)  Statements  are  immutable.  Note  that  the  content  of  Activities  that  are 
referenced  in  Statements  is  not  considered  part  of  the  Statement  itself.  So  while  the  Statement  is  immutable, 
the  Activities  referenced  by  that  Statement  are  not.  This  means  a  deep  serialization  of  a  Statement  into  JSON 
will  change  if  the  referenced  Activities  change  (see  the  Statement  API’s  “format”  parameter  for  details) 
(“Advanced  Distributed  Learning  Co-Laboratories,”  n.d..  Statement  properties  section). 
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Experience  API  provides  the  statement  structure  and  definition  which  allows  for 
statement  storage  (“Advanced  Distributed  Learning  Co-Laboratories,”  n.d.,  Role  of  the 
Experience  API  section).  The  statement  structure  enables  people  and  systems  to  conduct 
proper  analysis  of  the  collected  data.  Ligure  4  is  a  sample  learning  experience  captured  in 
xAPI  JSON  statement  format.  The  blue  boxes  depict  individual  properties  and  the  red 
boxes  depict  examples  of  the  types  of  data  available  for  analysis.  Each  property’s 
available  contents  and  format  will  vary.  A  complete  list  may  be  found  on  the  xAPI 
Statement  Data  Model  located  at  https://drive.google.eom/file/d/0BxhK5TH2E 
sphZFB  Xe  VNnS  Goz  WEE/vie  w . 
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Figure  4.  Captured  Learning  Experience  in  xAPI  JSON  Statement  Format. 


{  4 - Statement  Beginning 

"id" :  "7ef d071f-68be-4  96f-aa6f-6f 4cebaaea07 " , 
"verb":  { 


iy":  { 

"en-US":  "answered" 

^  Type  of  Learning 

} 

Experience 

}, 


"timestamp" :  "2016-04-26T17 : 15 : 16 . 3090002" , 
"object":  { 

"id":  "VM101 _ HYD_1 " . 

"objectType" :  "Activity" 

}, 


T 


Properties 


/ 


’actor":  ( 


"name":  "Student  1", 


" ob j  ectType " :  "Agent " 


Name  of 
Student 


"stored":  "2016-04-26T17 : 15 : 21 . 546919Z" , 


"result":  { 

:ore":  { 

Question 

"scaled":  1 

Score 

}, 

"response":  "VM101_HYD_1D" 

Properties 


1 


’context":  { 

"extensions":  { 

"http : / / easyaenerator/ expapi/ course/ id" :  "VM1011 

}, 

"contextActivities" :  { 


"parent 

[ 

f 

"id":  "LO  A", 

"objectType" : 

Learning  Objective 


>]}>,} 
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III.  PROTOTYPE  USE  CASES  FOR  DEVELOPMENT 


One  way  to  determine  learning  effectiveness  is  to  evaluate  a  student’s 
performance  of  specific  learning  activities.  To  do  this  requires  a  two-step  process.  The 
first  step  in  this  process  requires  linking  course  assessments  to  course  learning  objectives. 
The  second  step  requires  assessing  their  level  of  learning  by  analyzing  the  student’s 
performance  in  relation  the  course  learning  objectives.  This  thesis  focuses  on  the  first 
step  of  this  process  by  creating  a  prototype  system  with  the  infrastructure  and  courseware 
capable  of  capturing  and  storing  learning  experiences  as  xAPI  statements.  The  stored 
statements  will  contain  the  results  of  assessment  questions  and  their  associated  learning 
objectives  which  can  be  used  in  determining  individual  learning  objective  performances 
values.  Although  this  thesis  does  not  cover  the  second  step,  it  will  show  an  example  of 
data  extraction  from  the  repository  and  discuss  possible  future  research  in  assessing  the 
level  of  student  learning. 

Three  use  cases  were  developed  to  demonstrate  linking  course  assessments  to 
course  learning  objectives.  The  first  use  case  covers  course  structure  and  the  process  to 
create  an  online  course.  The  second  use  case  covers  the  technologies  (LMS,  LRS,  xAPI) 
used  to  capture  and  store  assessment  statements.  The  third  use  case  covers  the  extraction 
of  data  from  the  repository.  Since  e-leaming  courses  on  MarineNet  are  typically  self- 
paced  courses  with  automated  multiple  choice  assessments,  the  courseware  structure  and 
assessments  created  for  the  prototype  system  are  limited  in  scope  to  those  types  of 
courses.  For  background  purposes  only,  the  use  cases  cover  some  concepts  outside  of 
multiple  choice  assessments. 

A.  STAKEHOLDERS 

The  use  cases  cover  the  following  stakeholder  groups:  students,  course 
instructors,  and  course  developers.  Table  3  depicts  each  stakeholder  with  their  associated 
tasks.  This  thesis  refers  to  the  course  instructor  and  developer  as  separate  individuals  with 
separate  tasks  but  they  can  the  same  person  in  a  real  educational  setting.  A  fourth 
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stakeholder  group,  institutional  administrators,  is  listed  in  Table  3,  but  is  only  referenced 
briefly  in  Use  Case  3,  Chapter  IV,  and  Chapter  V. 


Table  3.  Stakeholders  and  Their  Tasks. 


Stakeholder 

Tasks 

Student 

•  request  enrollment  in  course  on  LMS 

•  take  course  and  assessments 

Course  Instructor 

•  compile  course  learning  objectives 

•  create  content  and  assessments  that  will  support  those 
learning  objectives 

•  identify  course  performance  data  to  be  captured  and 
stored 

•  enroll  or  approve  enrollment  request  of  student  in 
course  on  LMS 

•  analyze  results  of  student  learning  objective 
performances 

Course  Developer 

•  develop  course  on  an  LMS  given  course  content  and 
structure  from  course  instructor 

•  create  repository  for  performance  data  to  be  stored  on 

•  ensure  course  performance  data  is  captured  and 
stored 

•  extract  results  of  student  learning  objective 
performances  for  analysis  by  instructors 

Institutional 

Administrators 

•  identification  of  courses  and  their  goals  to  support 
curriculum 

•  analyze  results  of  student  learning  objective 

performances  to  evaluate  course  effectiveness  in 
curriculum  objectives 

•  analyze  results  of  student  learning  objective 

performances  to  evaluate  instructor  effectiveness  in 
course  and  curriculum  objectives 
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B.  USE  CASE  1— COURSWARE  CREATION 

The  Eberly  Center  at  Carnegie  Mellon  University  states  that  course  assessments 
should  capture  the  amount  a  student  has  learned,  and  for  this  to  occur  the  following  three 
components  must  be  aligned:  learning  objectives,  instructional  strategies,  and 
assessments  (“Why  should  assessments,”  n.d.).  They  also  define  instructional  strategy  as 
the  activities  that  will  reinforce  learning  activities  and  prepare  students  for  assessments 
(“Why  should  assessments,”  n.d.).  The  instructional  strategy  used  in  the  use  cases  and 
prototype  system  courseware  will  deliver  self-paced  course  content  with  automated 
multiple  choice  assessments.  This  strategy  closely  models  the  structure  of  current 
MarineNet  courses  offered. 

The  actors  in  Use  Case  1  are  the  course  instructor  and  developer.  Use  Case  1 
assumes  an  instructor  has  received  a  course  goal  which  establishes  the  overall  purpose 
and  outcome  of  the  course.  The  instructor’s  first  step  after  assessing  the  course  goal  is  to 
create  the  course  learning  objectives.  The  second  step  creates  assessment  questions  that 
properly  evaluate  a  student’s  knowledge  for  the  individual  course  learning  objectives. 
The  instructor’s  last  step  creates  structured  content  that  support  the  learning  objectives 
and  assessment  questions.  Lastly,  the  course  developer  takes  the  completed  course 
content  and  structure,  and  publishes  the  course  in  a  format  compatible  with  the  LMS. 

1.  Learning  Objectives 

Learning  objectives  are  statements  that  clearly  convey  what  a  student  should 
learn,  or  task  they  should  be  able  to  complete  after  a  given  period  of  instruction.  Learning 
objectives  are  important  to  a  course’s  development  and  structure  because  ensure  all 
aspects  of  the  course  goal  are  met  while  providing  organization  and  direction  for  the 
intended  course  topics.  According  the  Office  of  Distance  Learning  at  Llorida  State 
University,  instructors  should  create  learning  objectives  that  provide  a  clear 
understanding  of  the  desired  learning  outcomes  without  focusing  on  the  form  of 
instruction,  and  also  be  focused  on  student  performance  (2011).  The  following  are  several 
key  characteristics  all  learning  objectives  should  have:  specific,  measurable,  attainable, 
relevant,  and  targeted  at  the  student. 
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Three  main  components  of  learning  objectives  are  person,  verb,  and  conditions. 
The  person  component  is  who  the  learning  objective  is  intended  for,  typically  “the 
student.”  The  verb  component  is  the  action  the  person  is  to  accomplish.  The  conditions 
component  are  the  circumstances  the  person  conducts  the  verb  actions.  The  following  are 
examples  of  complete  learning  objectives. 

•  Given  a  diagram  of  a  passenger  vehicle,  the  student  locates  the  five  major 
hydraulic  brake  components  of  a  passenger  vehicle. 

•  Without  the  aid  of  reference,  the  student  describes  the  purpose  of  a 
vehicle’s  steering  system. 

2.  Course  Assessments 

A  course  assessment  is  a  tool  used  by  the  instructor  to  determine  the  learning 
objective  performance  of  a  student.  The  Eberly  Center  recommends  instructors  take  the 
following  into  consideration  when  creating  assessment  questions:  selection  of  an 
appropriate  question  type  to  match  the  learning  objective,  point  value  of  questions,  time 
allotted  for  assessment,  and  having  a  third  party  review  questions  (“Creating  Exams,” 
n.d.).  Instructors  also  need  to  ensure  the  content  and  scope  of  every  question  in  the 
assessment  directly  relates  to  a  course  learning  objective. 

A  few  question  types  that  can  be  asked  in  assessments  are  multiple-choice,  essay, 
short  answer,  and  matching.  It  is  important  the  question  type  supports  the  verb  in  the 
learning  objective.  If  a  learning  objective’s  verb  is  to  “explain”  then  an  appropriate 
question  type  to  evaluate  the  student’s  performance  would  be  an  essay  or  short  answer, 
not  a  multiple-choice.  The  question  type  needs  consideration  when  assigning  point 
values,  quantity  and  type  of  questions,  and  amount  of  time  allotted.  Essay  and  short 
answer  questions  typically  take  longer  for  a  student  to  respond  to  than  matching 
questions.  Lastly,  it  is  important  for  third  party  review  of  the  assessments  to  ensure 
question  and  answer  clarity  and  scope.  Since  this  thesis’  focus  is  multiple  choice 
assessments,  all  prototype  courseware  learning  objectives  are  supported  by  this  type  of 
question. 

When  creating  course  assessments,  instructors  must  ensure  each  learning 
objective  is  represented  in  the  assessment  by  the  minimum  number  of  questions  needed 
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to  assure  a  proper  evaluation  of  performance.  Typically,  multiple  assessment  questions 
are  asked  to  ensure  the  learning  objective  is  properly  assessed.  In  a  case  where  a  course 
goal  requires  a  high  number  of  learning  objectives  or  assessment  questions  to  evaluate  a 
student’s  performance,  the  use  of  in-course  quizzes,  standard  EOC  exams,  or  a 
combination  of  both  can  be  used.  This  technique  to  distribute  questions  throughout  a 
course  can  ensure  the  adequate  evaluation  of  all  learning  objective  performances  while 
maintaining  practical  quiz  and  EOC  exam  lengths. 

3.  Course  Content  and  Structure 

Course  instructors  can  choose  from  many  different  types  of  content  when 
deciding  how  best  to  support  the  course  learning  objectives  and  assessments.  Types  of 
content  can  include:  multimedia,  tables,  images,  documents,  and  plain  text.  Table  4 
provides  the  Center  for  Advanced  Teaching  and  Learning  at  the  University  of  Wisconsin 
La  Crosse’s  popular  ways  of  presenting  content. 


Table  4.  Popular  Content  Presentation  Types.  Adapted  from  “Ways  to 

present  content”  (n.d.). 


Presentation  Type 

Content  Type 

Characteristics 

Video  and  Audio 

Multimedia 

Tables 

Images 

•  good  for  demonstrations 

•  time  consuming 

•  best  if  accompanied  with  text 
version 

Portable  Document 
Format  (PDF) 

Tables 

Images 

Text 

Documents 

•  maintains  formatting  on  all  devices 

•  downloadable,  printable 

•  presents  text  with  accompanied 
images  well 

Slides  (PowerPoint) 

Tables 

Images 

Text 

•  consider  accompanying  with  audio 

website  Links 

Tables 

Images 

Text 

Documents 

Multimedia 

•  provides  access  to  other 

presentation  methods 

•  access  to  articles,  textbooks,  case 
studies 

•  requires  Internet  connection 
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Two  methods  to  structure  and  deliver  the  course  content  are  push  and  pull.  The 
push  method  delivers  course  content  in  a  sequential  order  determined  by  the  course 
instructor  and  requires  students  to  access  all  content  in  the  prior  sections/modules  prior  to 
moving  on  to  the  next  (see  Figure  5).  This  method  is  useful  for  instructors  to  ensure 
content  is  accessed  in  a  particular  order,  ensuring  compliance  of  content  access,  and  for 
ensuring  all  students  receive  the  same  course  experience. 


Figure  5.  Push  Method  of  Course  Content  Delivery.  Adapted  from 

Kuhlmann  (2009). 
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The  pull  method  of  delivering  course  content  allows  access  to  all  course  content 
and  relies  on  the  student  to  seek  out  the  information  they  feel  they  need  to  accomplish  the 
course  learning  objectives  (see  Figure  6).  This  method  allows  each  student  to  customize 
their  own  learning  experiences  and  creates  a  more  efficient  use  of  time  and  resources. 
The  pull  method  does  not  negate  the  need  to  conduct  an  assessment  to  determine  learning 
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objective  performances,  it  just  enables  students  to  focus  their  educational  efforts  on  the 
content  they  assess  they  need  based  on  previous  knowledge  and  experience. 


Figure  6.  Pull  Method  of  Course  Content  Delivery.  Adapted  from  Kuhlmann 

(2009). 
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4.  Publishing  a  Course 

In  the  context  of  e-learning,  publishing  a  course  has  a  few  different  meanings.  For 
the  purpose  of  this  thesis,  publishing  a  course  will  be  the  act  of  creating  a  SCORM  1.2  or 
2004  compliant  course  package  using  the  instructor  provided  course  elements  (learning 
objectives,  assessments,  and  content).  According  to  the  former  President  of  Rustici 
Software  Mike  Rustici,  the  SCORM  compliant  package  is  typically  a  ZIP  file  type 
containing  all  the  files  and  folders  needed  to  deliver  the  course  and  is  referred  to  as  a 
package  interchange  file  (PIF)  (2009).  The  PIF  is  what  allows  a  SCORM  compliant 
course  to  be  loaded  on  to  any  LMS  capable  of  supporting  that  version  of  SCORM. 
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The  course  developer  typically  uses  course  generator  or  authoring  tool  software  to 
upload  and  structure  the  elements  of  the  course  into  the  instructor’s  desired  format.  Once 
this  is  complete,  the  course  developer  will  select  the  required  version  of  SCORM  and  use 
the  software  to  publish  the  course.  According  to  Rustici,  when  publishing  a  course  the 
software  creates  and  places  into  the  PIF,  the  manifest  file,  Shareable  Content  Objects 
(SCOs),  assets,  and  other  SCORM  required  support  files  (2009).  A  SCO  is  a  unit  of 
instruction  that  contains  one  or  files  that  communicates  with  the  LMS  and  an  asset  is 
collection  of  one  or  more  static  content  files  that  is  presented  to  the  student  but  does  not 
communicate  with  the  LMS  (2009).  Rustici  also  states,  the  manifest  file  is  an  extensible 
Markup  Language  (XML)  file  that  describes  all  the  content  in  the  PIF  to  include  the 
SCOs  and  assets,  and  must  exist  at  the  root  (2009).  It  is  important  for  the  developer  to 
verify  the  versions  of  SCORM  the  LMS  is  capable  of  supporting  prior  to  publishing  to 
ensure  the  course  will  be  compatible  with  the  desired  system. 

Experience  API  can  be  incorporated  into  a  SCORM  compliant  course  before  or 
after  a  developer  publishes  the  course.  Some  course  generation  software  have  the  ability 
to  select  preset  experience  statements  to  capture  and  designate  a  LRS  to  which  to  store 
them  on.  These  statements  can  include  the  following:  when  a  course  or  module  is  started 
or  stopped,  whether  a  student  passed  or  failed  the  course  or  module,  and  whether  a 
student  answered  a  question  correctly  or  not.  If  the  developer  selects  these  options  prior 
to  publishing  a  course  the  xAPI  statement  code  and  associated  JavaScript  files  will  be 
created  and  included  in  the  PIF.  Incorporating  xAPI  into  a  PIF  is  possible  by  manually 
creating  JavaScript  files  and  entering  xAPI  statement  code  into  existing  files.  Whether 
incorporating  xAPI  by  using  course  generation  software  or  manual  input,  it  is  important 
to  include  the  correct  LRS  endpoint  (LRS  location)  and  authentication  credentials  to 
ensure  the  statements  are  stored  properly.  This  method  can  be  time  consuming  and 
requires  a  very  detailed  knowledge  of  SCORM  and  xAPI. 
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c. 


USE  CASE  2— CAPTURE  AND  STORAGE  OF  ASSESSMENTS 


Use  Case  2  discusses  the  implementation  of  xAPI  on  a  LMS  only  learning 
architecture,  the  administrative  setup  of  the  course  PIF  on  the  LMS  course,  the  setup  of 
the  LRS,  and  how  a  student’s  interaction  with  a  course  creates  and  stores  xAPI 
statements.  The  actors  for  Use  Case  2  consist  of  the  course  developer  who 
administratively  sets  up  the  LMS  course  and  uploads  the  courseware  PIFs,  the  instructor 
who  verifies  the  course  represents  the  desired  elements  and  structure  as  well  as  enrolls 
the  student,  and  the  student  who  accesses  the  course  and  assessments. 

ADL  has  developed  the  SCORM-to-TLA  (Training  and  Learning  Architecture) 
Roadmap  which  is  a  four  phased  approach  for  transitioning  from  a  LMS  only  structure  to 
a  TLA  (“SCORM  to  TLA  Roadmap,”  n.d.).  Figure  7  is  a  graphical  depiction  of  ADL’s 
SCORM-to-TLA  Roadmap  with  descriptions  of  the  four  phases.  With  CDET’s 
MarineNet  architecture  currently  in  the  SCORM  phase  (Phase  0),  this  use  case  and  the 
prototype  system  will  only  discuss  the  setup  and  use  of  the  LMS  Centric  phase  (Phase  1). 


Figure  7.  ADL’s  SCORM-to-TLA  Roadmap.  Adapted  from  “SCORM  to 

TLA  Roadmap”  (n.d.). 
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Typical  SCORM  learning 
environment.  An  LMS  is  the  central 
hub  of  the  learning  environment, 
delivering  content  and  recording 
learner  progress. 

The  LMS  continues  to  be  the  central 
hub.  but  learner  progress  is  also  sent 
to  a  LRS  via  xAPI.  The  dual  tracking 
of  learner  experiences  provides  a 
consistent  means  to  access  that  data 
for  uses  such  as  reports  and 
analytics. 

The  reliance  on  the  LMS  has 

diminished.  The  LRS  stores  learner 
experiences  and  content  no  longer 
needs  to  be  managed  by  the  LMS.  At 
this  point,  the  LMS.  if  used,  is  just  a 
consumer  of  the  xAPI  service 

A  services  based  approach  to  training 
and  learning  environments.  APIs  for 
learning  services  are  defined  and 
used  to  create  a  a  completely 
customizable  training  and  learning 
environment. 

The  LMS  Centric  phase  uses  the  idea  of  “dual-tracking”  which  maintains  the 
LMS  as  the  center  of  the  learning  architecture  as  in  the  SCORM  phase,  but  implements 
xAPI  into  the  SCORM  compliant  course  to  add  the  ability  to  track  learning  experiences 
on  a  LRS  (“LMS  Centric,”  n.d.).  ADL  also  states  that  the  LMS  Centric  phase  has  the 


25 


benefit  of  allowing  an  organization  to  protect  and  maintain  its  existing  e- 
leaming/SCORM  knowledge  and  architecture,  while  providing  the  ability  to  track 
learning  experiences  a  LMS  is  incapable  of  tracking  (“LMS  Centric,”  n.d.). 
Implementing  a  LMS  Centric  architecture  allows  an  organization  to  phase  in  the  use  of 
xAPI  over  time  by  allowing  them  to  update  as  many  or  as  few  courses  as  the  course 
developers  can  handle.  This  would  allow  CDET  to  incrementally  update  their  courseware 
to  include  xAPI  without  disrupting  MarineNet’s  current  course  completion  tracking  while 
meeting  their  goal  of  finding  ways  to  improve  the  level  of  e-leaming  they  are  providing 
students. 

1.  LMS 

Three  roles  users  can  be  assigned  on  a  LMS  are  student,  instructor,  and 
administrator.  Most  LMS  have  the  ability  to  customize  the  permissions  for  each  of  user 
or  user  role.  To  not  cause  confusion  between  institution  administrators  and  the  role  of  a 
LMS  administrator,  this  use  case  will  assume  the  course  developer  has  been  assigned  the 
role  of  LMS  administrator. 

For  a  course  developer  to  upload  a  PIF  to  a  FMS,  a  FMS  course  must  first  be 
administratively  established.  This  allows  the  assignment  of  users  to  instructor  and  student 
roles  of  the  course,  enrollment/disenrollment,  course  status,  tracking  to  the  FMS  only, 
dates  of  course  availability,  and  the  upload  of  content  to  include  the  PIF.  Once  the  course 
developer  uploads  the  PIF  and  verifies  the  SCORM  compliant  course  is  running  properly 
on  the  FMS,  a  course  instructor  can  be  assigned  and  student  enrolled. 

2.  LRS 

Storing  xAPI  statements  to  an  FRS  requires  established  credentials  within  the 

administrative  settings  of  the  FRS.  Two  types  of  credentials  used  for  FRSs  are  Hypertext 

Transfer  Protocol  (HTTP)  Basic  Authentication  and  OAuth  2.0.  HTTP  Basic 

Authentication  is  a  method  of  authentication  that  requires  the  user  to  send  the  FRS’ 

credentials  (user  name  and  password)  in  the  clear  over  the  Internet.  This  provides  no 

confidentiality  of  the  credentials  and  poses  a  security  risk.  Request  for  Comments  (RFC) 

6749  created  by  the  Internet  Engineering  Task  Force,  states  that  OAuth  2.0  provides  a 
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more  secure  means  of  accessing  LRS  data  because  the  user  no  longer  uses  the  LRS’ 
credential  and  instead  is  issued  an  access  token  which  specifies  the  scope,  lifetime,  and 
other  access  attributes  (Internet  Engineering  Task  Force,  2012).  Figure  8  shows  the 
interaction  between  the  client  (user)  and  resource  server  (FRS). 


Figure  8.  OAuth  2.0  Abstract  Protocol  Flow.  Source:  Internet  Engineering 

Task  Force  (2012). 
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Similar  to  an  FMS,  an  FRS  usually  has  the  ability  to  assign  users  varying  roles  or 
levels  of  permission.  These  permissions  allow  the  statements  stored  to  the  FRS  to  be 
accessed  and  viewed  only  by  authorized  individuals.  They  can  also  be  used  to  control  the 
export  of  statements  off  the  FRS  to  other  systems  capable  to  using  them  in  further 
analysis. 


3.  Dual  Tracking  with  xAPI 

Dual  tracking  of  data  in  the  FMS  Centric  phase  requires  the  ability  to 
communicate  data  to  both  the  FMS  and  FRS  (Poltrack  &  Creighton,  2015).  Figure  9 
depicts  the  communication  relationships  between  the  FMS,  FRS,  and  the  SCORM  course 
with  xAPI  in  a  dual  tracking  configuration.  According  to  Poltrack  and  Creighton’s  paper, 
requested  content  containing  xAPI  JavaScript  code  is  sent  by  the  FMS  accompanied  by  a 
content  player  consisting  of  a  SCORM  API  which  facilitates  the  communication  of  data 
between  the  FMS  and  content  (2015).  When  the  content  is  ready  to  send  data  to  the  FMS 
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it  makes  Initialize()  and  SetValue()  calls  to  the  SCORM  API  while  simultaneously 
calling  functions  within  the  JavaScript  code  (APIWrapper.js  in  Figure  9)  (Poltrack  & 
Creighton,  2015).The  JavaScript  code  create  and  transmit  JSON  statements  to  the  LRS 
using  abstracted  SCO  tracked  data.  Although  xAPI  code  extracting  SCO  tracked  data  can 
be  created  manually,  ADL  developed  JavaScript  files  (i.e.,  APIWrapper.js, 
xapiwrapper.min.js,  SCORMToXAPIFunctions.js)  to  take  some  coding  complexities  out 
of  determining  the  LMS  interface  calls  needed  for  the  abstraction,  creation,  and 
transmission  of  data  to  the  LRS  (Poltrack  &  Creighton,  2015). 


Figure  9.  SCORM  Course  with  “Dual  Tracking”  to  LMS  and  LRS.  Source: 
Jonathan  Poltrack,  personal  communication,  March  14,  2016. 
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D.  USE  CASE  3— EXTRACTION  OF  DATA 


Use  Case  3  discusses  the  extraction  of  data  collected  from  the  learning 
experiences  captured  within  SCORM  courseware  integrating  xAPI.  The  actors  for  Use 
Case  3  are  students,  course  instructors,  course  developers,  and  institutional 
administrators.  Depending  on  the  types  of  data  collected  and  stored  in  the  LRS,  all  actors 
might  have  an  interest  in  accessing  various  parts  of  the  stored  statements.  The  following 
are  examples  of  uses  various  stakeholders  might  want  to  access  the  statements  for. 

•  Students — determine  their  own  performance  and  trends  over  a  subject  or 
curriculum 

•  Course  instructors — gather  learning  information  about  current  and  future 
students  to  customize  the  courses  they  deliver  in  order  to  provide  a  more 
effective  learning  environment 

•  Course  developers — determine  new  uses  of  the  current  data  collected  as 
well  as  determine  new  useful  data  to  collect 

•  Institutional  administrators — determine  trends  in  student  and  instructor 
performance  at  the  course  and  curriculum  level 

Three  ways  to  extract  data  from  an  LRS  are  pulling  the  data  off  the  LRS,  pushing 
the  data  out  of  the  LRS  to  a  system  capable  of  accepting  it,  and  exporting  the  captured 
JSON  statements  to  CSV  files.  This  prototype  system  exports  captured  JSON  statements 
to  CSV  files  due  to  the  universal  nature  of  the  CSV  file  format  to  be  accepted  on  a  wide 
range  of  systems  to  include  analytic  software  and  databases. 
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IV.  PROTOTYPE  SYSTEM  IMPLEMENTATION 


The  purpose  of  the  prototype  system  is  to  demonstrate  the  capture  and  storage  of 
student  learning  experiences  in  JSON  format  using  xAPI  in  a  LMS  Centric/dual-track 
architecture.  The  following  three  high-level  tasks  were  required  to  develop  the  prototype 
system:  creation  of  a  LRS  with  credentials  to  store  the  JSON  statements,  creation  of 
SCORM  compliant  courseware,  and  the  creation  of  a  LMS  to  administer  the  courseware. 
The  prototype  system  enables  a  collective  analysis  of  individual  learning  experience 
statements  allowing  learning  objective  proficiencies  to  be  calculated  into  a  student’s 
learning  objective  performance  value.  This  learning  objective  performance  value  can  then 
be  used  in  further  analysis  by  educators  to  make  a  more  detailed  assessment  as  to  the 
level  of  learning  a  student  has  attained  in  a  given  subject. 

A.  PROTOTYPE  ARCHITECTURE 

The  prototype  system  uses  a  LMS  Centric/dual-track  architecture  which  requires  a 
LMS,  LRS,  SCORM  compliant  course  configured  with  xAPI,  and  a  student  device  with 
web  browser.  Figure  10  depicts  the  prototype  dual-track  architecture  as  well  as  the  third- 
party  vendors  used  to  provide  the  required  services.  The  prototype  system  uses  Talent 
LMS  to  manage  the  administrative  tasks  of  managing  user  roles,  hosting  of  the  SCORM 
courseware  (VM101 — Passenger  Vehicle  Maintenance  Course,  VM201 — Commercial 
Vehicle  Maintenance  Course),  and  providing  SCORM  content  to  the  student’s  device. 
Saltbox  Wax  LRS  is  used  to  receive  and  store  the  JSON  statements  created  on  the 
student’s  device  through  the  interaction  with  the  SCORM  courseware.  Since  this 
architecture  is  using  third-party  vendors  to  provide  services,  it  assumes  continuous 
Internet  connectivity  between  the  student’s  device  and  LMS  to  allow  content, 
assessments,  and  course  results  to  be  communicated  back  and  forth.  Internet  connectivity 
between  the  student’s  device  and  LRS  is  only  required  during  the  transmission  of  JSON 
statements  to  the  LRS. 
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Figure  10.  Prototype  System  Architecture. 


*VM101  and  VM201  are  SCORM-compliant  courses  with  xAPI  created  using  easygenerator.com  and 
uploaded  to  Talent  LMS  for  course  hosting. 


B.  VENDOR  SERVICES 

Three  SaaS  vendors  were  used  to  create  the  prototype  system.  The  vendors  were 
selected  based  on  their  services  meeting  the  requirements  needed  to  establish  a  dual-track 
architecture.  Table  5  provides  a  list  of  the  third-party  vendors  and  their  services  used  in 
the  creation  of  the  prototype  system. 
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Table  5.  Third-Party  Vendors  and  Services  Provided/Utilized. 


Vendor 

Service  Plan 

Services  Provided/Utilized 

Easygenerator 

Plus  ($39.99  /  mo) 

•  learning  objective  creation 

•  course  delivery  layout  creation 
(pull  method  used) 

•  integration  of  xAPI 

•  quiz  and  exam  question  creation 
(single  choice  answer  only  used) 

•  publish  prototype  course  in 
SCORM  1.2 

•  publish  prototype  course  to 
Easygenerator  web  server  (testing 
only) 

•  create  up  to  100  courses  or  exams 
(2x  courses,  2x  exams  created) 

Talent  LMS 

Free  (free) 

•  5  user  limit  (4  users  created) 

•  10  course  limit  (2  courses  hosed) 

•  ability  to  assign  administrator, 
instructor,  and  student  roles 

•  manage  assignment  of  instructors 
and  enrollment  of  students 

•  assignment  of  courses  to  categories 

•  ability  to  upload  multiple  content 
formats  to  created  courses  (only 
SCORM  w/  xAPI  uploaded) 

•  manage  student’s  course 

completion  status 

Saltbox  Wax  LRS 

Explore  (free) 

•  2,000  JSON  statement  limit  per 
month 

•  ability  to  access  created  JSON 
statements 

•  ability  to  create  Basic  and  OAuth 
credentials 
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C.  COMPONENTS 

1.  Courseware 

Creating  the  prototype  courseware  required  a  two-step  process.  The  first  step 
developed  the  course  elements  (learning  objectives,  assessments,  and  content).  The 
second  step  created  a  SCORM  PIF  with  xAPI  capable  of  extracting  JSON  statements  to 
the  designated  LRS  using  the  course  elements.  The  courseware  PIFs  were  then  be 
uploaded  to  an  LMS  allowing  users  to  administer  and  interact  with  the  courses. 

a.  Learning  Objective,  Assessments,  Content 

The  prototype  courseware  consists  of  one  curriculum  program  with  two  individual 
courses.  Each  course  has  four  modules  with  quizzes  and  contains  an  EOC  exam.  Each 
module  is  linked  to  one  learning  objective  and  provides  the  student  course  content  and 
two  quiz  questions  directly  relating  to  that  learning  objective.  The  EOC  exam  consists  of 
two  questions  per  learning  objective  for  a  total  of  eight  questions.  Table  6  outlines  each 
course,  its  learning  objective,  and  parent  subject  of  each  learning  object.  The  full  course 
outline  including  all  modules,  learning  objectives,  content,  and  quiz  and  exam  assessment 
questions  with  all  possible  answers  can  be  referenced  in  Appendix  A. 
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Table  6.  Prototype  Courseware  Learning  Objectives  and  Parent  Subject. 


Course 

Parent  Subject 

Learning  Objectives 

VM101 

Brakes 

Learning  Objective  (A) — Without  the  aid  of 
reference,  recall  the  major  components  of  a 
vehicle’s  hydraulic  brake  system  and  their  purpose. 

Electrical 

Learning  Objective  (B) — Without  the  aid  of 
reference,  recall  the  major  components  of  a  vehicle 
electrical  svstem  and  their  purpose. 

Steering 

Learning  Objective  (C) — Without  the  aid  of 
reference,  recall  the  major  components  of  a 
vehicle’s  steering  svstem  and  their  purpose. 

Engines 

Learning  Objective  (D) — Without  the  aid  of 
reference,  recall  the  major  components  of  a 
gasoline  engine  and  their  purpose. 

VM201 

Brakes 

Learning  Objective  (E) — Without  the  aid  of 
reference,  recall  the  major  components  of  a 
vehicle’s  air  brake  svstem  and  their  purpose. 

Electrical 

Learning  Objective  (B) — Without  the  aid  of 
reference,  recall  the  major  components  of  a  vehicle 
electrical  svstem  and  their  purpose. 

Steering 

Learning  Objective  (C) — Without  the  aid  of 
reference,  recall  the  major  components  of  a 
vehicle’s  steering  svstem  and  their  purpose. 

Engines 

Learning  Objective  (F) — Without  the  aid  of 
reference,  recall  the  major  components  of  a  diesel 
engine  and  their  purpose. 

Curriculum — Vehicle  Maintenance,  VM101 — Passenger  Vehicle  Maintenance  Course,  VM201 — 
Commercial  Vehicle  Maintenance  Course. 


The  two  prototype  courses  have  shared  learning  objectives  (B  and  C)  and  similar 
learning  objectives  (A  and  E,  D  and  F)  that  are  subsets  of  a  parent  subject  in  order  to 
emphasize  the  possible  relationships  between  different  courses  within  a  curriculum. 
These  relationships  along  with  scores  of  the  individual  learning  objective  questions  are 
used  in  the  analysis  of  the  JSON  statements  and  the  calculation  of  the  learning  objective 
performance  value. 
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b.  Publishing  SCORM  PIF 

The  SCORM  PIFs  in  the  prototype  courseware  were  published  in  SCORM  1.2 
format  using  the  services  provided  at  Easygenerator.com.  Each  course  consists  of  two 
PIFs.  The  first  PIF  contains  the  modules  with  associated  learning  objectives  while  the 
second  PIF  contains  the  EOC  exam  questions  and  associated  learning  objectives.  Each 
module  (referred  to  as  sections  in  Easygenerator.com)  contains  the  content  and  quiz 
questions  associated  with  only  one  learning  objective.  Figure  11  depicts  VMlOl’s 
Hydraulic  Brake  System  Module’s  content,  two  quiz  questions,  and  linked  learning 
objective.  The  remaining  modules  in  both  courses  are  structured  the  same  as  Figure  11. 
The  EOC  exams  for  both  courses  are  structured  the  same  as  Figure  1 1  without  the  content 
item. 


Figure  11.  Easygenerator.com  Course  Generator  Software  Module/Feaming 

Object  Example. 
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/  VM101-  Passenger  Vehicle  Maintenance  Course 

Simple  course 


i»0  |  <*>  el 

Co-authors  Preview  Share 


1.  CREATE 


(e)  Section 


2.  DESIGN  >  3.  CONFIGURE  >  4.  PUBLISH  >  5.  RESULTS 


(A)  Without  the  aid  of  reference,  recall  the  major  components  of  a  vehicle's  hydraulic 
brake  system  and  their  purpose. 


Related  items 


Hydraulic  Brake  System  Content 

Last  edited:  04/1 5/201 6 


Which  of  the  following  is  NOT  a  major  component  of  a  vehicle's  hydraulic  brake  system? 

Last  edited:  04/08/2016 

Which  of  the  following  describes  the  purpose  of  the  hydraulic  lines? 

Last  edited:  02/1 1/2016 


New  item 


o 

e 

e 


Example  of  the  courseware’s  module  layout  linking  of  the  learning  objective  to  content 
and  quiz  questions.  Screenshot  from  http://live.easygenerator.eom/#eourses/ 
ee912cd399f8463a8b088cdf77785a77/sections/c01461 1 1 171b4d62ba835ca91ea3e25c. 
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The  prototype  courseware  uses  a  pull  content  delivery  method  which  allows  the 
student  to  complete  the  modules  and  learning  objectives  in  the  order  they  feel  best 
supports  their  learning.  Figure  12  depicts  VMlOl’s  design  layout  using  the  pull  content 
delivery  method. 


Figure  12.  Easygenerator.com  Course  Generator  Software  Content  Delivery. 
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Course:  "VM-101  -  Passenger  Vehicle  Maintenance 
Course" 


Submit  results 


Take  a  break  and  continue  later 


(A)  Without  the  aid  of  reference,  recall  the  major  components  of  a  vehicle's  hydraulic  brake 
system  and  their  purpose 

©  0% 
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(B)  Without  the  aid  of  reference,  recall  the  major  components  of  a  vehicle's  electrical  system 
and  their  purpose 

©  0% 


Start 


Start 


fr- 


(C)  Without  the  aid  of  reference,  recall  the  major  components  of  a  vehicle  s  steering  system 
and  theu  purpose 

O  0% 


Start 


(D)  Without  the  aid  of  reference,  recall  the  major  components  of  a  gasolme  engme  and  their 
fif#  purpose 

©  0% 


Start 


Example  of  the  courseware’s  pull  content  delivery  method  setup  and  design  allowing  a  student 
to  “Start”  the  module  of  their  choice.  Screenshot  from  http://live.easygenerator.eom/#courses/ 
ee912cd399f8463a8b088cdf77785a77/design. 


The  prototype  courseware  uses  xAPI  to  capture  and  transmit  to  the  LRS  how  a 
student  answered  a  question  and  whether  they  passed  or  failed  the  EOC  exam.  An  xAPI 
JSON  statement  is  created  for  every  quiz  and  exam  question  a  student  submits,  and  at  the 
completion  of  an  exam.  Figure  13  depicts  Easygenerator.com’ s  LRS  configuration  page 
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requiring  a  LRS  Uniform  Resource  Locator  (URL),  login  with  password  (credentials), 
and  requested  experience  statements.  The  actions  the  courseware  PIFs  track  are  answered 
statements  (individual  question  results)  for  quiz  questions  in  modules,  and  answered  and 
passed/failed  statements  (percentage  of  exam  questions  correct  or  incorrect)  are  tracked 
in  EOC  exams.  The  prototype  courseware  uses  a  percentage  value  of  75%  to  determine 
EOC  passed/failed  statements. 


Figure  13.  Easygenerator.com  Course  Generator  Software  xAPI  Tracking 

Settings. 


ADVANCED  SETTINGS  a 

Report  to: 

Custom  LRS  settings 

easygenerator  (recommended) 

LRS  URL 

https://cmacalon.waxlrs.com/TCAPI/statements 

Q  Custom  LRS 

Q  Authentication  required 

LAP  login 

NNy9HpzyNzxbVljKWtmR 

LAP  password 

8Q82AsLcxKnqKfpZloho 

Use  statements: 

Started 

Stopped  Passed 

Answered 

Mastered  Progressed 

Expirenced 

Failed 

Example  of  the  courseware’s  LRS  configuration  settings.  Screenshot  from 
http://live.easygenerator.eom/#courses/ee912cd399f8463a8b088cdf77785a77/configure. 


When  publishing  the  prototype  courseware  PIFs,  Easygenerator.com’ s  software 
assigns  unique  identification  numbers  in  the  form  of  16-byte  hexadecimal  values  to  each 
course  title,  module  title,  learning  objective,  quiz  and  exam  question,  and  each  answer 
option  to  the  questions.  These  identification  numbers  are  used  in  the  answered  and 
passed/failed  JSON  statements  sent  to  the  LRS.  The  identification  number  assignments 
are  saved  as  objects  within  a  JavaScript  file  (data.js)  contained  in  the  PIF.  Since  each  PIF 
was  published  individually,  the  same  learning  objective  was  assigned  multiple  unique 
identification  numbers  when  appearing  in  multiple  PIFs.  Each  learning  objective  appears 
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in  a  minimum  of  two  PIFs,  a  module  and  EOC  exam  PIF.  Learning  objective  B  and  C 
appear  in  all  four  PIFs  since  they  are  found  in  both  VM101  and  VM201  (see  Table  6, 
Prototype  Courseware  Learning  Objectives  and  Parent  Subject). 

To  ensure  the  JSON  statements  stored  in  the  LRS  can  be  grouped  by  course  and 
learning  objective,  each  course  title  and  learning  objective  unique  identification  number 
within  the  courseware  were  modified  to  facilitate  the  determination  of  a  student’s 
learning  objective  performance  value.  Also,  to  make  the  JSON  statements  easier  to  read 
and  understand,  as  well  as  demonstrate  the  ability  to  customize  the  xAPI  statements,  the 
module  titles,  questions,  and  answer  identification  numbers  were  modified  to  a  user- 
friendly  naming  convention.  Table  7  uses  VMlOl’s  Hydraulic  Brake  Module  to 
demonstrate  naming  convention  used  throughout  the  courseware  data.js  files. 


Table  7.  Courseware  Identification  Renaming  Convention. 


Element 

Title 

Renamed 

Identification 

Course  Name 

Passenger  Vehicle  Maintenance  Course 

VM101 

Learning  Objective 

(A)  Without  the  aid  of  reference,  recall 
the  major  components  of  a  vehicle’s 
hydraulic  brake  system  and  their  purpose. 

LO_A 

Module  Name 

Hydraulic  Brake  System  Module 

VM101_HYD 

First  Question  of 

Module 

Which  of  the  following  is  NOT  a  major 
component  of  a  vehicle’s  hydraulic  brake 
system? 

VM101_HYD_1 

Answer  choice  (a)  to 
Question  1 

brake  pedal 

VM101_HYD_1A 

c.  x API  Statement  Properties 

The  xAPI  statements  created  by  the  prototype  courseware  contain  the  same 
structure  and  objects  that  can  be  found  in  Figure  3  (Chapter  II — Top  Level  xAPI 
Statement  Data  Model)  except  version  and  attachment  properties.  Tables  8  and  9  provide 
complete  lists  of  properties  found  in  the  two  types  of  xAPI  statements  the  courseware 
creates,  answered  and  passed/failed  respectively.  The  primary  difference  between  the  two 
statements  types  is  that  the  individual  question  properties  (object.definition)  can  be  found 
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in  the  answered  statements.  Since  answered  statements  are  created  by  a  student’s 
submission  of  an  answer  to  a  question,  all  applicable  question  properties  are  added  to  the 
xAPI  statement.  The  passed/failed  statement  will  not  contain  these  properties  as  this 
statement  is  only  created  at  the  submission  and  scoring  of  an  entire  exam. 


Table  8.  Prototype  xAPI  Answered  Statement  Properties  with  Description. 


Property  Elements 

Description 

verb,  id 

Identification  of  action  statements 

verb. display 

Title  of  action  statements  (answered) 

timestamp 

Date  and  time  statement  was  created 

object. definition.choices. id 

Identification  of  question’s  answer  choice 

obj  ect  .definition  .choices .  description 

Title  of  question’s  answer  choice 

object.definition.correctResponsesPattem 

Identification  of  question’s  correct  response 

object.definition.type 

Type  of  SCORM  interaction  extracted  to 
xAPI 

object.definition.name 

Question  text 

object.definition.interactionType 

Question  type 

object.id 

Identification  of  question 

obj  ect  .obj  ectT ype 

Object  type  of  the  question 

actor.mbox 

Email  address  of  the  student 

actor,  name 

Name  of  the  student 

actor.objectType 

Object  type  of  student 

stored 

Date  and  time  statement  is  stored  to  LRS 

result. score. scaled 

Numerical  value  of  response 

Answered  =  0  or  1 

result.response 

Students  response  to  question 

context.registration 

Unique  identification  given  all  context  in  a 
course 

context .  extensions  .http :  //eas  ygenerator/e 
xpapi/course/id 

Identification  of  Course 

context .  context  Activities  .parent . 
definition.name 

Text  of  learning  objective 

context. contextActivities.parent. id 

Identification  of  learning  objective 

context. contextActivities.parent. 
objectType 

Object  type  of  learning  objective 

id 

Unique  identification  give  to  statement 

authority.  account.homePage 

URL  of  LRS 

authority .  account  .name 

Username  of  account  on  LRS 

authority  .obj  ectType 

Object  type  account 
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Table  9.  Prototype  xAPI  Passed/Failed  Statement  Properties  with 

Description. 


Property  Elements 

Description 

verb,  id 

Identification  of  action  statements 

verb,  display 

Title  of  action  statements  (passed  or  failed) 

object.definition.name 

Title  of  Exam 

object.id 

Identification  of  exam  title 

obj  ect  .obj  ectType 

Object  type  of  the  exam 

actor.mbox 

Email  address  of  the  student 

actor,  name 

Name  of  the  student 

actor.objectType 

Object  type  of  student 

stored 

Date  and  time  statement  is  stored  to  LRS 

result. score. scaled 

Numerical  value  of  response 

Passed/Failed  =  0  through  1  (decimal  value 
for  SCORM  1.2) 

context.registration 

Unique  identification  given  all  context  in  a 
course 

context .  extensions  .http :  //eas  ygenerator/e 
xpapi/course/id 

Identification  of  Course 

id 

Unique  identification  give  to  statement 

authority. account.homePage 

URL  of  LRS 

authority .  account  .name 

Username  of  account  on  LRS 

authority  .obj  ectType 

Object  type  account 

2.  LMS  Configuration 

The  prototype  system  uses  a  cloud  based  LMS  provided  by  TalentLMS  to  manage 
the  courseware  administration  and  delivery.  Configuring  the  prototype  LMS  consists  of 
the  creation  of  users  and  their  roles,  a  curriculum,  and  courses.  For  the  purpose  of  this 
thesis  an  LMS  course  is  defined  as  the  administrative  object  on  an  LMS  to  which  users 
are  assigned,  course  completion  within  the  LMS  tracked,  the  location  courseware  PIFs 
are  uploaded  to,  and  where  the  PIF  launch  settings  are  configured. 

a.  LMS  Users 

Four  users  were  created  and  assigned  one  of  three  user  roles  within  the  LMS 
providing  them  varying  levels  of  permissions  to  conduct  their  required  actions.  The 
administrator  user  role  allows  a  user  to  create  and  delete  users,  user  types,  courses, 
assignment  of  instructors,  and  manage  all  administrative  settings  within  the  LMS.  The 


41 


instructor  user  role  allows  a  user  to  add  courses  and  content,  enroll  students,  and  view 
course  completion  status  and  results.  The  student  user  role  allows  a  student  to  request 
enrollment  in  a  course  and  take  a  course.  Table  10  provides  a  complete  list  of  users,  user 
role  assigned,  and  actions  conducted. 


Table  10.  LMS  Users  and  Actions. 


User 

LMS  User  Role 

Actions 

Course  Developer 

Administrator 

•  Create  users 

•  Create  curriculum  Vehicle 

Maintenance 

•  Create  courses  VM101  and  VM201 

•  Assign  Instructor 

Course  Instructor 

Instructor 

•  Upload  courseware  PIFs 

•  Enroll  Student_l  and  Student_2  in 
both  VM101  and  VM201 

•  Track  student  course  completion 

Student_l 

Feamer 

•  Take  VM101  and  VM201  courses 

Student_2 

Feamer 

•  Take  VM101  and  VM201  courses 

b.  Curriculum  and  Courses 

The  course  category  Vehicle  Maintenance  was  created  to  maintain  organization 
within  the  LMS,  simulate  a  larger  overarching  course  catalog,  and  act  as  a  curriculum  of 
courses  a  student  must  complete.  The  LMS  courses  VM101  and  VM201  were  created 
within  the  Vehicle  Maintenance  category  and  the  courseware  PIFs  were  uploaded  as 
SCORM  content  to  their  respective  LMS  courses.  Each  LMS  course  contains  a 
module/quiz  PIF  and  an  EOC  exam  PIF.  PIF  launch  settings  within  the  FMS  course  are 
set  to  enforce  that  only  the  EOC  exam  must  be  taken  in  order  for  FMS  course 
completion.  The  setting  not  requiring  the  module/quiz  PIF  to  be  completed  allows  a 
student  the  option  to  proceed  directly  to  the  EOC  exam  if  they  feel  their  knowledge  in  the 
learning  objectives  is  sufficient  enough  to  meet  the  standards  assigned  by  the  course 
instructor. 
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3. 


LRS  Configuration 


The  configuration  of  the  prototype  system’s  LRS  consists  of  establishment  of  a 
LRS  URL  and  a  single  set  of  basic  credentials.  Only  one  set  of  LRS  credentials  are 
required  in  the  prototype  system  because  both  VM101  and  VM201  courses  are  instructed 
by  the  same  individual. 

D.  SEQUENCE  DIAGRAMS 

The  complete  prototype  system  with  its  courseware  produces  and  transmits  xAPI 
statements  during  student’s  interaction  with  the  module/quiz  and  EOC  Exam.  In  both  the 
module/quiz  and  EOC  Exam,  a  student  device  with  Internet  browser  is  required  in  order 
for  the  courseware  to  be  launched  and  viewed,  answers  to  questions  submitted,  and  xAPI 
statements  transmitted  to  the  LRS.  Since  the  LMS  course  contains  both  the  module/quiz 
and  EOC  exam  the  student  device-to-LMS  interaction  to  launch  either  is  identical.  The 
module/quiz  and  EOC  exam  differ  in  timing  of  the  construction,  submission,  and 
transmission  of  xAPI  statements  to  the  LRS.  Figures  14  and  15  are  sequence  diagrams  for 
a  student  taking  the  module/quiz  or  EOC  exam  and  both  assume  the  student  is  a  logged 
into  the  LMS  and  enrolled  in  the  LMS  course. 

1.  Module  with  Quiz  Submission 

As  shown  in  Figure  14,  the  student  device  requests  (HTTP  get)  the  LMS  course 

from  the  LMS  which  replies  (HTTP  post)  with  the  menu  allowing  the  student  to  select 

the  module/quiz  (HTTP  get).  The  LMS  responds  to  the  selection  the  module/quiz  by 

sending  the  content  player  for  initialization  on  the  student  device  (see  Figure  9  in  Chapter 

II,  Section  C  for  detailed  explanation  of  the  content  player).  With  the  content  player 

initialized  on  student  device,  all  requests  and  replies  for  content  (HTML)  and  quiz 

questions  (HTML/JavaScript)  occur  through  the  content  player.  The  module/quiz 

displays  one  question  at  a  time  for  submission.  When  the  student  device  submits  a  quiz 

answer  to  the  content  player,  the  content  player  calculates  the  question  score  and 

constructs  an  answered  xAPI  statement  in  JSON  format  with  the  properties  found  in 

Table  8.  The  content  player  then  sends  the  LMS  the  question  result  and  the  LRS  the 

constructed  answered  xAPI  statement.  The  process  of  requesting  content  and  quiz 
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question  will  continue  until  the  request  by  the  student  device  to  close  the  module/quiz. 
The  closure  request  is  sent  to  through  the  content  player  to  the  LMS  at  which  time  the 
content  player  is  deconstructed  and  deleted  from  the  student  device.  The  final  step  in  the 
process  consists  of  the  LMS  sending  to  the  student  device  the  LMS  course  menu  allowing 
the  student  to  select  either  the  module/quiz  or  EOC  exam. 


Figure  14.  Student  Module/Quiz  Sequence  Diagram.  Adapted  from  Jonathan 
Poltrack,  personal  communication,  March  14,  2016. 
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2. 


EOC  Exam  Submission 


As  shown  in  Figure  15,  the  student  device  requests  (HTTP  get)  the  LMS  course 
from  the  LMS  which  replies  (HTTP  post)  with  the  menu  allowing  the  student  to  select 
the  EOC  Exam  (HTTP  get).  The  LMS  responds  to  the  selection  the  EOC  Exam  by 
sending  the  content  player  for  initialization  on  the  student  device  (see  Figure  9  in  Chapter 
II,  Section  C  for  detailed  explanation  of  the  content  player).  Once  the  content  player  is 
initialized  on  student  device,  the  request  for  exam  questions  (HTML/JavaScript)  occurs 
through  the  content  player.  The  EOC  Exam  displays  all  exam  questions  on  the  same  page 
and  has  a  single  submission  for  all  answers  simultaneously.  When  the  student  device 
submits  the  exam  question  answers  to  the  content  player,  the  content  player  calculates  the 
result  of  each  question  score  individually  and  constructs  answered  xAPI  statements  in 
JSON  format  with  the  properties  found  in  Table  8.  The  content  player  uses  the  results  of 
the  exam  question  scores  to  calculate  a  total  exam  score  and  construct  a  passed  or  failed 
xAPI  statement  in  JSON  format  with  the  properties  found  in  Table  9.  The  content  player 
then  sends  the  LMS  the  exam  result  and  the  LRS  the  constructed  answered  and 
passed/failed  xAPI  statements.  When  the  student  completes  with  the  EOC  Exam  the 
closure  request  is  sent  through  the  content  player  to  the  LMS  at  which  time  the  content 
player  is  deconstructed  and  deleted  from  the  student  device.  The  final  step  in  the  process 
consists  of  the  LMS  sending  to  the  student  device  the  LMS  course  menu  allowing  the 
student  to  select  either  the  module/quiz  or  EOC  exam. 
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Figure  15.  Student  EOC  Exam  Sequence  Diagram.  Adapted  from  Jonathan 
Poltrack,  personal  communication,  March  14,  2016. 
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E.  LEARNING  OBJECTIVE  PERFORMANCE  VALUE 

The  purpose  of  the  prototype  system  is  to  demonstrate  the  capture  and  storage  of 
student  learning  experiences  in  JSON  format.  The  learning  experiences  are  then  used  to 
calculate  learning  objective  performance  values  which  can  then  assist  educators  in  the 
improvement  of  the  education  they  deliver.  A  learning  objective  performance  value  is  a 
percentage  of  questions  pertaining  to  a  single  or  group  of  learning  objectives  answered 
correctly  by  one  or  more  students.  To  calculate  a  learning  objective  performance  value, 
the  total  number  of  questions  answered  correctly  is  divided  by  the  total  number  of 
questions  answered  and  this  decimal  value  is  then  converted  to  a  percentage.  Learning 
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objective  performance  values  calculated  in  this  thesis  are:  single  student/single  learning 
objective,  single  student/multiple  learning  objectives  (parent  subject  of  multiple  learning 
objectives),  multiple  students/single  learning  objective,  and  multiple  students/multiple 
learning  objectives. 

1.  Administer  of  Prototype  Courseware 

The  demonstration  of  the  prototype  system  used  two  simulated  students 
(Student_l  and  Student_2)  to  complete  all  portions  of  the  module/quiz  and  EOC  exam 
for  both  VM101  and  VM201.  The  student  responses  to  the  module/quiz  and  EOC  exam 
questions  were  scripted  in  order  to  register  course  completion  within  the  LMS  and 
generate  xAPI  statements  that  provide  specific  calculated  learning  objective  performance 
values.  Each  student’s  completion  of  VM101  and  VM201  updated  the  LMS  course 
completion  data  with  exam  scores  and  registered  the  student  as  passing  the  course  (see 
Figure  16).  The  question  responses  of  both  simulated  students  created  and  stored  64 
answered  statements  and  4  passed  statements  to  the  prototype  system’s  LRS.  The  passed 
statements  generated  duplicate  the  LRS  course  completion  status  recorded  within  the 
LMS  course.  A  complete  list  of  Student_l  and  Student_2  responses  to  each  question  in 
the  courseware  and  the  EOC  exam  scores  can  be  found  in  Appendix  B. 
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Figure  16.  LMS  Course  Completion  with  Score. 
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Instructors  view  of  VM101  LMS  completion  report  depicting  Student_l  and  Student_2. 

passing  the  course  and  providing  the  results  of  the  EOC  exams.  Screenshot  from 

https  ://cmacthesis .  talentlms  .com/reports/ scormtestinfo/id :  1 792 . 

2.  Calculation  of  the  Learning  Objective  Performance  Values 

There  are  four  property  elements  needed  from  the  answered  statements  in  order  to 
calculate  learning  objective  performance  values:  actor.name  (Student_#),  object.id 
(Question  ID),  result. score  (Correct  or  Incorrect),  and  context. contextActivities. parent. id 
(Learning  Objective  ID).  To  obtain  these  property  elements,  the  64  xAPI  answered 
statements  in  JSON  format  were  retrieved  from  the  LRS,  converted  to  a  CSV  file,  and 
saved  as  a  Microsoft  Excel  spreadsheet.  The  learning  objective  performance  values  in 
Figures  17  and  18  were  calculated  using  the  values  the  four  property  elements  above 
contained  within  the  Microsoft  Excel  spreadsheet. 

a.  Single  Student  /  Single  and  Multiple  Learning  Objectives 

The  single  student  /  single  learning  objective  performance  values  provide  a 
student’s  performance  by  individual  learning  objective  rather  than  focusing  on  their  exam 
performance  at  the  end  of  a  course.  How  well  a  student  scores  on  exams  or  quizzes  is 
typically  how  the  learning  success  of  a  student  is  equated  too.  Figure  17  shows  each 
student  passing  VM101  and  VM201  EOC  exams  with  scores  of  75%  or  greater.  Also 
depicted  is  each  student’s  individual  learning  objective  performance  values  and  the 
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number  of  questions  answered  correctly  to  the  total  number  of  questions.  Analyzing  the 
individual  learning  objective  performance  values  shows  that  even  though  both  Student_l 
and  Student_2  passed  both  courses,  they  each  performed  poorly  with  regard  to  specific 
learning  objectives  (Student_l  with  62.5%  in  learning  objective  B  and  Student_2  with 
50%  in  learning  objective  E). 

The  single  student  /  multiple  learning  objective  performance  values  breakdown  a 
student’s  performance  by  the  parent  subject  the  individual  learning  objectives  belong  to. 
A  collective  analysis  of  learning  objectives  can  show  how  a  well  a  student  performs  with 
regard  to  particular  subjects  within  a  curriculum.  Learning  objectives  A  and  E  are 
examples  of  how  hydraulic  brakes  and  air  brakes  would  belong  to  the  same  parent  subject 
of  brakes.  Figure  17  depicts  Student_2’s  poor  performance  in  the  subject  of  brakes  while 
scoring  high  in  the  other  subjects. 


Figure  17.  Student’s  Individual  Learning  Objectives  Performance  Values. 
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Learning  objective  performance  values — single  student/single  learning  objective,  single  student/multiple 
learning  objectives  (brakes,  electrical  steering,  engines).  Right  column — number  of  questions  correct  /  total 
number  of  questions. 


b.  Multiple  Students  /  Single  and  Multiple  Learning  Objectives 

The  multiple  students  /  single  and  multiple  learning  objective  performance  values 


provide  a  group  of  students’  collective  performance  with  regard  to  individual  learning 
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objectives  and  parent  subjects.  An  analysis  of  both  students’  EOC  exam  results  would 
yield  a  vehicle  maintenance  curriculum  exam  average  of  86%.  This  analysis  alone  would 
not  identify  the  poor  collective  performance  values  of  Student_l  and  Student_2  in  the 
learning  objective  E  and  the  parent  subject  of  brakes  (see  Figure  18). 


Figure  18.  Combined  Student’s  Individual  Learning  Objectives  Performance 

Values. 
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Learning  objective  performance  values — multiple  students/single  learning 
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steering,  engines).  Right  column — number  of  questions  correct  /  total 
number  of  questions. 
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3.  Use  of  Learning  Objective  Performance  Values 

Learning  objective  performance  values  can  be  used  by  students,  instructors,  and 
administrators  to  determine  where  areas  of  education  and  study  should  be  focused.  They 
help  capture  a  more  detailed  representation  of  how  well  a  student  or  group  of  students  is 
performing,  and  whether  the  instruction  or  curriculum  is  effective.  Analyzing  learning 
objective  performance  values  over  time  can  assist  in  determining  trends  in  students’ 
learning,  effectiveness  of  new  teaching  materials  or  methods,  or  success  of  updated 
curriculums. 

Students  can  use  their  own  learning  objective  performance  values  to  determine 
where  their  strengths  and  weaknesses  may  lie  and  use  that  information  to  determine  more 
efficient  methods  of  studying.  Students  may  also  use  them  to  select  courses  within  a 
curriculum  they  feel  provide  them  the  most  useful  amount  education  for  what  they  intend 
on  using  it  for.  Over  time,  a  student  can  use  the  trends  of  their  learning  objective 
performance  values  to  assess  the  effectiveness  of  the  adjustments  they  made  in  their 
education. 

Instructors  can  analyze  the  learning  objective  performance  values  of  their  students 
to  tailor  the  instruction  they  are  delivering.  By  tailoring  their  courses,  instructors  can 
maximize  the  instruction’s  efficiency  by  focusing  less  time  on  areas  where  students  excel 
and  more  time  where  they  struggle.  Instructors  can  use  their  students’  learning  objective 
performance  value  trends  to  make  assessments  as  to  the  continued  effectiveness  of  the 
materials,  content,  assessment  questions,  or  teaching  methods  they  are  delivering. 

Institutional  administrators  can  use  collective  learning  objective  performance 
values  to  assist  in  determining  curriculum  decisions.  Analyzing  the  collective  learning 
objectives  within  a  curriculum  can  assist  administrators  in  making  decisions  as  to  where 
educational  resources  (time,  space,  and  money)  should  be  allocated  to  provide  the  most 
effective  education  to  their  students. 
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V.  CONCLUSIONS  AND  FUTURE  RESEARCH 


A.  CONCLUSIONS 

The  prototype  system  demonstrates  a  method  CDET  can  use  to  integrate  xAPI 
into  their  courseware  while  capturing  relevant  learning  experiences.  These  learning 
experiences  are  capable  of  providing  valuable  data  leading  to  improvement  in  the 
effectiveness  and  efficiency  of  their  education.  The  LMS  Centric  architecture  provides 
CDET  the  benefit  of  maintaining  their  existing  infrastructure,  with  the  addition  of  an 
LRS,  while  allowing  them  to  focus  on  updating  courseware  with  xAPI.  This  architecture 
also  affords  CDET  the  benefit  of  not  disrupting  the  MarineNet’s  current  course  delivery 
and  tracking  and  allows  an  opportunity  for  other  educational  systems  to  provide  data  to 
the  LRS  for  additional  analysis.  The  limitation  with  MarineNet  established  in  a  LMS 
Centric  architecture  is  its  continued  reliance  on  SCORM  compliant  courses  to  deliver  e- 
leaming  programs.  Marines’  access  to  smart  devices  worldwide,  with  or  without 
connectivity  to  a  network,  provide  an  opportunity  to  take  advantage  of  xAPIs  ability  to 
capture  learning  experiences  outside  the  traditional  student  device-to-LMS  connection. 

Learning  objective  performance  values  provide  a  quantitative  metric  that  can 
enable  CDET  to  begin  assessing  the  amount  of  proficiency  a  student  obtains  in  individual 
learning  objectives,  subjects,  or  curriculums.  The  use  of  learning  objective  performance 
values  will  allow  CDET  to  begin  placing  specific  values  on  the  proficiencies  of  the 
learning  objectives  currently  maintained  throughout  all  their  courseware  providing  a 
more  detailed  measurement  of  the  intended  results  of  each  course.  Students’  strengths 
and  weaknesses  in  specific  areas  can  also  be  determined  and  used  to  create  tailored 
curriculums  that  will  allow  time  and  resources  to  be  spent  in  subject  areas  that  will  most 
benefit  the  student  and  the  Marine  Corps.  CDET  can  also  use  learning  objective 
performance  values  to  help  determine  the  relevancy  and  effectiveness  of  specific  course 
content,  learning  objectives,  and  course  assessments.  The  limitations  with  the  learning 
objective  performance  values  discussed  in  this  thesis  are  that  they  were  calculated  using 
only  assessment  questions  from  self-paced,  non-instructor  evaluated  e-leaming  courses. 
More  learning  experiences  from  different  types  of  assessments  (evaluated  practical 
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applications,  essay  questions,  fill-ins,  and  matching)  need  to  be  calculated  into  the 
learning  objective  performance  values  to  make  them  a  more  accurate  representation  of  a 
student’s  proficiency  in  a  subject  area.  The  analysis  CDET  can  conduct  using  learning 
objective  performance  values  and  the  changes  they  can  enact  from  that  analysis  can  lead 
them  to  improving  the  effectiveness  and  efficiency  of  their  e-leaming  program. 

B.  FUTURE  RESEARCH 

The  prototype  system  demonstrated  in  this  thesis  was  limited  in  scope  to  the 
capture  and  storage  of  learning  experiences  from  basic  SCORM  compliant  courses  and 
the  calculation  of  learning  objective  performance  values.  Future  research  in  the  following 
topics  is  recommended  to  enhance  the  value  of  the  methods  and  ideas  proposed  in  this 
thesis. 


1.  Curriculum  Learning  Objective  Analysis 

This  thesis  created  two  courses  with  limited  learning  objectives,  content,  and 
assessments  to  demonstrate  how  xAPI  could  assist  in  the  calculation  of  learning  objective 
performance  values.  The  next  step  would  apply  the  concept  of  the  overlapping  learning 
objectives  and  parent  subjects  to  an  entire  curriculum  of  courses.  Applying  this  concept 
would  allow  educators  to  create  a  taxonomy  of  learning  objectives  providing  the 
relationships  necessary  to  produce  the  most  valuable  learning  objective  performance 
values  to  their  organization. 

•  How  would  a  learning  objective  with  a  subject  of  “hydraulic  brakes”  be  a 
related  to  the  parent  subjects  of  brakes,  hydraulics,  passenger  vehicles, 
commercial  vehicles,  and  construction  vehicles? 

•  How  do  learning  objectives  over  multiple  curriculums  throughout  an 
entire  career  relate  to  each  other?  How  can  learning  objective  performance 
value  trends  throughout  a  career  be  calculated  and  used  in  the  analysis? 
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2. 


LRS  Interaction 


The  interaction  with  the  LRS  in  this  thesis  was  limited  to  pulling  raw  JSON 
statements  off  the  LRS  and  converting  them  to  a  CSV  file.  LRSs  provide  varying  levels 
of  analytic  tools  that  can  assist  a  user  with  providing  some  level  of  data  analysis. 

•  What  other  platforms  can  the  LRS  experience  statements  be  exported  to? 
Do  those  platforms  have  more  capable  analytic  tools? 

•  What  are  the  best  ways  for  non-technical  individuals  (students  and 
instructors)  to  interact  with  the  data?  What  data  is  best  presented  (the 
entire  experience  statement  or  elements  of  the  statement)?  What  formats 
(web  application,  documents,  forms)  should  be  used  to  present  the  data? 

3.  Capture  of  Learning  Experiences 

The  two  learning  experiences  captured  in  this  thesis  were  the  scores  of  the 
question  answers  and  the  passed/failed  course  status  with  score  as  they  both  relate  to  a 
learning  objective.  These  learning  experiences  provide  valuable  data  but  are  only  two  of 
many  others  occurring  while  a  student  interacts  with  an  e-leaming  course. 

•  What  student-course  interactions  are  occurring  and  which  should  be 
captured  and  analyzed? 

•  Beside  SCORM  courses,  what  other  methods  of  instruction  are  occurring 
within  an  organization  which  xAPI  could  be  used  to  capture  learning 
experiences? 

•  What  other  forms  of  assessments  (evaluated  practical  applications,  essay 
questions,  fill-ins)  should  be  captured  and  how  should  they  be  weighed 
against  the  others? 
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APPENDIX  A.  PROTOTYPE  COURSES 


A.  VM101— PASSENGER  VEHICLE  MAINTENANCE  COURSE 

Hydraulic  Brake  System  Module 

Learning  Objective  A  -  Without  the  aid  of  reference,  recall  the  major 
components  of  a  vehicle’s  hydraulic  brake  system  and  their  purpose. 

Content  -  The  major  components  of  a  vehicle’s  hydraulic  brake  system  are  the 
brake  pedal,  pushrod,  master  cylinder  assembly,  hydraulic  brake  lines,  and  the 
brake  caliper  assembly.  The  purpose  of  the  hydraulic  brake  lines  is  to  carry 
brake  fluid  from  the  master  cylinder  to  the  brake  caliper  assembly.  The  purpose 
of  the  brake  pedal  is  to  transfer  the  driver’s  input  requesting  to  slow  the  speed  of 
a  vehicle  to  a  mechanical  action  by  the  entire  hydraulic  brake  system. 

Module  Quiz  Questions  (Learning  Objective  A) 

Which  of  the  following  is  NOT  a  major  component  of  a  vehicle’s 
hydraulic  brake  system? 

a)  brake  pedal 

b)  pushrod 

c)  hydraulic  brake  lines 

d)  steering  wheel 

Which  of  the  following  describes  the  purpose  of  the  hydraulic  lines? 

a)  carry  the  windshield  wiper  fluid 

b)  carry  the  brake  fluid  from  the  master  cylinder  to  the  brake 
caliper  assembly 

c)  carry  the  electrical  current  from  the  ignition  to  the  starter 

d)  open  the  vehicle  doors 

Electrical  System  Module 

Learning  Objective  B  -  Without  the  aid  of  reference,  recall  the  major 
components  of  a  vehicle  electrical  system  and  their  purpose. 

Content  -  The  major  components  of  a  vehicle  electrical  system  are  the  engine 
control  unit,  alternator,  battery,  starter  solenoid,  starter,  fuse  box,  ignition 
switch,  and  the  starter  relay.  The  purpose  of  the  alternator  is  to  recharge  the 
battery  of  the  power  it  uses  while  the  vehicle  is  running.  The  purpose  of  the 
ignition  switch  is  to  activate  the  electrical  power  of  a  vehicle.  The  purpose  of  the 
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battery  is  to  provide  the  electrical  current  needed  to  start  the  vehicle.  The 
purpose  the  starter  solenoid  is  to  convert  electrical  current  to  mechanical  motion 
to  enable  the  starter  motor  to  start  the  vehicle. 

Module  Quiz  Questions  (Learning  Objective  B) 

Which  of  the  following  is  NOT  a  major  component  of  a  vehicle’s 
electrical  system? 

a)  fuse  box 

b)  battery 

c)  tire 

d)  alternator 

Which  of  the  following  describes  the  purpose  of  the  alternator? 

a)  recharge  the  battery 

b)  inflate  the  tires 

c)  open  the  doors 

d)  cool  the  engine  down 

Steering  System  Module 

Learning  Objective  C  -  Without  the  aid  of  reference,  recall  the  major 
components  of  a  vehicle’s  steering  system  and  their  purpose. 

Content  -  The  major  components  of  a  vehicle  steering  system  are  the  steering 
wheel,  steering  column,  power  steering  pump,  powering  steering  fluid  reservoir, 
tie  rods,  rack  and  pinon  unit,  and  tires.  The  purpose  of  the  steering  column  is  to 
connect  the  steering  wheel  to  the  rack  and  pinon  unit  to  transfer  the  driver  inputs 
to  the  remaining  steering  assembly.  The  purpose  of  the  tie  rods  is  to  connect  the 
steering  assembly  to  the  wheels  and  ensure  the  wheels  maintain  alignment.  The 
purpose  of  the  power  steering  pump  is  to  provide  the  hydraulic  fluid  pressure 
needed  to  operate  the  power  steering  system.  The  purpose  the  power  steering 
fluid  reservoir  is  to  store  hydraulic  fluid  needed  to  operate  the  power  steering 
system. 

Module  Quiz  Learning  Objective  C 

Which  of  the  following  is  NOT  a  major  component  of  a  vehicle’s  steering 
system? 

a)  passenger  door 

b)  power  steering  pump 

c)  steering  wheel 

d)  steering  column 
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Which  of  the  following  describes  the  purpose  of  the  steering  column? 

a)  transfer  driver  inputs  to  remaining  steering  assembly 

b)  inflate  the  tires 

c)  carry  the  electrical  current  from  the  ignition  to  the  starter 

d)  cool  the  engine  down 

Gasoline  Engine  Module 

Learning  Objective  D  -  Without  the  aid  of  reference,  recall  the  major 
components  of  a  gasoline  engine  and  their  purpose. 

Content  -  The  major  components  of  a  gasoline  engine  are  the  crankshaft, 
camshaft,  timing  belt,  fuel  injector,  intake  manifold,  exhaust  manifold,  cylinder 
head  cover,  pistons,  engine  block,  spark  plugs,  electric  coil,  distributor,  and  oil 
pan.  The  purpose  of  the  crankshaft  is  to  convert  the  up  and  down  motion  of  the 
pistons  into  rotational  motion  which  enable  the  turning  of  the  drive  train.  The 
purpose  of  the  fuel  injector  is  to  provide  the  proper  fuel  air  mixture  to  the 
engine’s  combustion  chambers  to  enable  movement  of  the  pistons. 

Module  Quiz  Learning  Objective  D 

Which  of  the  following  is  NOT  a  major  component  of  a  gasoline  engine? 

a)  exhaust  manifold 

b)  distributor 

c)  timing  belt 

d)  steering  column 

Which  of  the  following  describes  the  purpose  of  the  crankshaft? 

a)  recharge  the  battery 

b)  convert  the  up  and  down  motion  of  the  pistons  into  rotational 
motion 

c)  increase  the  speed  of  the  vehicle 

d)  cool  the  engine  down 

Passenger  Vehicle  Maintenance  EOC  Exam 
Learning  Objective  A  Questions 

Which  of  the  following  is  NOT  a  major  component  of  a  vehicle’s 
hydraulic  brake  system? 

a)  master  cylinder  assembly 

b)  brake  caliper  assembly 

c)  tires 
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d)  brake  pedal 

Which  of  the  following  describes  the  purpose  of  the  brake  pedal? 

a)  open  the  windows 

b)  open  the  trunk 

c)  transfer  the  driver’s  input  to  slow  the  speed  of  the  vehicle  into 
a  mechanical  action  by  the  entire  hydraulic  brake  system 

d)  increase  the  speed  of  the  vehicle 

Learning  Objective  B  Questions 

Which  of  the  following  is  NOT  a  major  component  of  a  vehicle’s 
electrical  system? 

a)  alternator 

b)  trunk 

c)  fuse  box 

d)  starter  solenoid 

Which  of  the  following  describes  the  purpose  of  the  ignition  switch? 

a)  activate  the  electrical  power  of  a  vehicle 

b)  adjust  the  vehicle  seats 

c)  assist  in  latching  the  seat  belt 

d)  store  additional  fuses 

Learning  Objective  C  Questions 

Which  of  the  following  is  NOT  a  major  component  of  a  vehicle’s  steering 
system? 

a)  steering  wheel 

b)  alternator 

c)  power  steering  pump 

d)  rack  and  pinon  unit 

Which  of  the  following  describes  a  purpose  of  the  tie  rods? 

a)  activate  the  electrical  power  of  a  vehicle 

b)  recharge  the  battery 

c)  ensure  the  wheels  maintain  alignment 

d)  open  the  windows 

Learning  Objective  D  Questions 

Which  of  the  following  is  NOT  a  major  component  of  a  gasoline  engine? 
a)  camshaft 
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b)  timing  belt 

c)  power  steering  pump 

d)  intake  manifold 

Which  of  the  following  describes  the  purpose  of  the  fuel  injector? 

a)  provide  the  proper  fuel  air  mixture  to  the  engine’s  combustion 
chambers 

b)  ensure  the  wheels  maintain  alignment 

c)  activate  the  electrical  power  of  a  vehicle 

d)  transfer  the  driver’s  input  to  slow  the  speed  of  the  vehicle  into 
a  mechanical  action  by  the  entire  hydraulic  brake  system 

B.  VM201— COMMERCIAL  VEHICLE  MAINTENANCE  COURSE 

Air  Brake  System  Module 

Learning  Objective  E  -  Without  the  aid  of  reference,  recall  the  major 
components  of  a  vehicle’s  air  brake  system  and  their  purpose. 

Content  -  The  major  components  of  a  vehicle’s  air  brake  system  are  the 
compressor,  main  reservoir,  auxiliary  reservoir,  air  dryer,  brake  pedal,  brake 
pipe,  drain  valves,  and  hand  brake.  The  purpose  of  the  compressor  is  to  pump  air 
into  the  reservoirs  to  be  used  in  the  operation  of  the  air  brake  system.  The 
purpose  of  the  main  reservoir  is  to  store  pressurized  air  to  be  used  in  the 
operation  of  the  air  brake  system. 

Module  Quiz  Questions  (Learning  Objective  E) 

Which  of  the  following  is  NOT  a  major  component  of  vehicle’s  air  brake 
system? 

a)  brake  pedal 

b)  main  reservoir 

c)  hydraulic  brake  lines 

d)  air  dryer 

Which  of  the  following  describes  the  purpose  of  the  compressor? 

a)  pump  air  into  the  reservoirs 

b)  open  the  vehicle  doors 

c)  carry  the  electrical  current  from  the  ignition  to  the  starter 

d)  carry  the  brake  fluid  from  the  master  cylinder  to  the  brake 
caliper  assembly 

Electrical  System  Module 
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Learning  Objective  B  -  Without  the  aid  of  reference,  recall  the  major 
components  of  a  vehicle  electrical  system  and  their  purpose. 

Content  -  The  major  components  of  a  vehicle  electrical  system  are  the  engine 
control  unit,  alternator,  battery,  starter  solenoid,  starter,  fuse  box,  ignition  switch, 
and  the  starter  relay.  The  purpose  of  the  alternator  is  to  recharge  the  battery  of 
the  power  it  uses  while  the  vehicle  is  running.  The  purpose  of  the  ignition  switch 
is  to  activate  the  electrical  power  of  a  vehicle.  The  purpose  of  the  battery  is  to 
provide  the  electrical  current  needed  to  start  the  vehicle.  The  purpose  the  starter 
solenoid  is  to  convert  electrical  current  to  mechanical  motion  to  enable  the  starter 
motor  to  start  the  vehicle. 

Module  Quiz  Questions  (Learning  Objective  B) 

Which  of  the  following  is  NOT  a  major  component  of  a  commercial 
vehicle’s  electrical  system? 

a)  starter 

b)  starter  relay 

c)  solenoid 

d)  main  reservoir 

Which  of  the  following  describes  the  purpose  of  the  battery? 

a)  inflate  the  tires 

b)  adjust  the  vehicle  seats 

c)  carry  hydraulic  fluid  from  the  hydraulic  reservoir  to  the  brake 
caliper 

d)  provide  the  electrical  current  needed  to  start  the  vehicle 

Steering  System  Module 

Learning  Objective  C  -  Without  the  aid  of  reference,  recall  the  major 
components  of  a  vehicle’s  steering  system  and  their  purpose. 

Content  -  The  major  components  of  a  vehicle  steering  system  are  the  steering 
wheel,  steering  column,  power  steering  pump,  powering  steering  fluid  reservoir, 
tie  rods,  rack  and  pinon  unit,  and  tires.  The  purpose  of  the  steering  column  is  to 
connect  the  steering  wheel  to  the  rack  and  pinon  unit  to  transfer  the  driver  inputs 
to  the  remaining  steering  assembly.  The  purpose  of  the  tie  rods  is  to  connect  the 
steering  assembly  to  the  wheels  and  ensure  the  wheels  maintain  alignment.  The 
purpose  of  the  power  steering  pump  is  to  provide  the  hydraulic  fluid  pressure 
needed  to  operate  the  power  steering  system.  The  purpose  the  power  steering 
fluid  reservoir  is  to  store  hydraulic  fluid  needed  to  operate  the  power  steering 
system. 

Module  Quiz  Learning  Objective  C 
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Which  of  the  following  is  NOT  a  major  component  of  a  commercial 
vehicle’s  steering  system? 

a)  tie  rods 

b)  drain  valves 

c)  rack  and  pinon  unit 

d)  steer  fluid  reservoir 

Which  of  the  following  describes  the  purpose  of  the  power  steering  pump? 

a)  provide  the  hydraulic  fluid  pressure  needed  to  operate  the 
power  steering  system 

b)  transfer  driver  inputs  to  remaining  steering  assembly 

c)  cool  the  engine  down 

d)  carry  the  electrical  current  from  the  ignition  to  the  starter 

Diesel  Engine  Module 

Learning  Objective  F  -  Without  the  aid  of  reference,  recall  the  major 
components  of  a  diesel  engine  and  their  purpose. 

Content  -  The  major  components  of  a  diesel  engine  are  the  crankshaft,  camshaft, 
timing  belt,  fuel  injector,  intake  manifold,  exhaust  manifold,  cylinder  head  cover, 
pistons,  engine  block,  and  oil  pan.  The  purpose  of  the  piston  is  to  transfer  the 
energy  caused  by  the  explosion  of  the  fuel/air  mixture  within  the  combustion 
chamber  to  the  crankshaft.  The  purpose  of  the  camshaft  is  to  convert  its 
rotational  motion  to  a  up  and  down  motion  through  the  push  rods  and  rocker 
arms  to  open  valves  into  the  combustion  chambers. 

Module  Quiz  Learning  Objective  F 

Which  of  the  following  is  NOT  a  major  component  of  a  diesel  engine? 

a)  engine  block 

b)  distributor 

c)  spark  plug 

d)  fuel  injector 

Which  of  the  following  describes  the  purpose  of  the  piston? 

a)  transfer  the  energy  caused  by  the  explosion  of  the  fuel/air 
mixture  within  the  combustion  chamber  to  the  crankshaft 

b)  increase  the  speed  of  the  vehicle 

c)  convert  the  up  and  down  motion  of  the  pistons  into  rotational 
motion 

d)  cool  the  engine  down 

Commercial  Vehicle  Maintenance  EOC  Exam 
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Learning  Objective  E  Questions 

Which  of  the  following  is  NOT  a  major  component  of  a  vehicle’s  air  brake 
system? 

a)  master  cylinder  assembly 

b)  hand  brake 

c)  drain  valves 

d)  auxiliary  reservoir 

Which  of  the  following  describes  the  purpose  of  the  main  reservoir? 

a)  transfer  the  driver’s  input  to  slow  the  speed  of  the  vehicle  into 
a  mechanical  action  by  the  entire  hydraulic  brake  system 

b)  adjust  the  vehicle’s  seats 

c)  open  the  trunk 

d)  store  pressurized  air 

Learning  Objective  B  Questions 

Which  of  the  following  is  NOT  a  major  component  of  a  vehicle’s 
electrical  system? 

a)  auxiliary  reservoir 

b)  alternator 

c)  fuse  box 

d)  starter  solenoid 

Which  of  the  following  describes  the  purpose  of  the  starter  solenoid? 

a)  store  additional  fuses 

b)  convert  electrical  current  to  mechanical  motion  to  enable  the 
starter  motor  to  start  the  vehicle 

c)  assist  in  latching  the  seat  belt 

d)  store  pressurized  air 

Learning  Objective  C  Questions 

Which  of  the  following  is  NOT  a  major  component  of  a  vehicle’s  steering 
system? 

a)  steering  column 

b)  fuel  injector 

c)  tires 

d)  rack  and  pinon  unit 

Which  of  the  following  describes  a  purpose  of  the  power  steering  fluid 
reservoir? 

a)  ensure  the  wheels  maintain  alignment 

b)  activate  the  electrical  power  of  a  vehicle 
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c)  recharge  the  battery 

d)  store  hydraulic  fluid  needed  to  operate  the  power  steering 
system 

Learning  Objective  F  Questions 

Which  of  the  following  is  NOT  a  major  component  of  a  diesel  engine? 

a)  exhaust  manifold 

b)  compressor 

c)  timing  belt 

d)  oil  pan 

Which  of  the  following  describes  the  purpose  of  the  camshaft? 

a)  open  the  trunk 

b)  convert  its  rotational  motion  to  a  up  and  down  motion  through 
the  push  rods  and  rocker  arms  to  open  valves  into  the 
combustion  chambers 

c)  activate  the  electrical  power  of  a  vehicle 

d)  transfer  the  driver’s  input  to  slow  the  speed  of  the  vehicle  into 

a  mechanical  action  by  the  entire  hydraulic  brake  system 
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APPENDIX  B.  COURSEWARE  QUESTION  RESPONSES 


Student  Response — 1  indicates  a  correct  response,  0  indicates  an  incorrect 
response. 


VM101  Module/Quiz 

Type  of 
Statement 

Learning 

Obj 

Learning  Obj 
Subject 

Question 

Student_l 

Response 

Student_2 

Response 

Answered 

LO_A 

Hydraulic  Brakes 

VM101_HYD_1 

1 

0 

Answered 

LO_A 

Hydraulic  Brakes 

VM101_HYD_2 

1 

1 

Answered 

LO_B 

Electrical 

VM101_ELE_1 

0 

1 

Answered 

LO_B 

Electrical 

VM101_ELE_2 

0 

1 

Answered 

LO_C 

Steering 

VM101_STE_1 

1 

1 

Answered 

LO_C 

Steering 

VM101_STE_2 

1 

1 

Answered 

LO_D 

Gasoline  Engines 

VM101_GAS_1 

1 

1 

Answered 

LO_D 

Gasoline  Engines 

VM101_GAS_2 

1 

1 

VM101  EOC  Exam 

Type  of 
Statement 

Learning 

Obj 

Learning  Obj 
Subject 

Question 

Student_l 

Response 

Student_2 

Response 

Answered 

LO_A 

Hydraulic  Brakes 

VM101_EXAM_1 

0 

1 

Answered 

LO_A 

Hydraulic  Brakes 

VM 10 1_EXAM_2 

1 

1 

Answered 

LO_B 

Electrical 

VM101_EXAM_3 

1 

1 

Answered 

LO_B 

Electrical 

VM101_EXAM_4 

1 

1 

Answered 

LO_C 

Steering 

VM101_EXAM_5 

1 

1 

Answered 

LO_C 

Steering 

VM  10 1_EXAM_6 

1 

1 

Answered 

LO_D 

Gasoline  Engines 

VM101_EXAM_7 

1 

1 

Answered 

LO_D 

Gasoline  Engines 

VM101_EXAM_8 

1 

1 
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VM201  Module/Quiz 

Type  of 
Statement 

Learning 

Obj 

Learning  Obj 
Subject 

Question 

Student_l 

Response 

Student_2 

Response 

Answered 

LO_E 

Air  Brakes 

VM201_AIR_1 

1 

0 

Answered 

LO_E 

Air  Brakes 

VM201_AIR_2 

1 

1 

Answered 

LO_B 

Electrical 

VM201_ELE_1 

1 

1 

Answered 

LO_B 

Electrical 

VM201_ELE_2 

1 

1 

Answered 

LO_C 

Steering 

1 

1 

Answered 

LO_C 

Steering 

1 

1 

Answered 

LO_F 

Diesel  Engines 

VM201_DIE_1 

0 

1 

Answered 

LO_F 

Diesel  Engines 

VM201_DIE_2 

1 

1 

VM201  EOC  Exam 

Type  of 
Statement 

Learning 

Obj 

Learning  Obj 
Subject 

Question 

Student_l 

Response 

Student_2 

Response 

Answered 

LO_E 

Air  Brakes 

VM201_EXAM_1 

0 

0 

Answered 

LO_E 

Air  Brakes 

VM  20 1_EXAM_2 

1 

1 

Answered 

LO_B 

Electrical 

VM201_EXAM_3 

0 

1 

Answered 

LO_B 

Electrical 

VM201_EXAM_4 

1 

1 

Answered 

LO_C 

Steering 

1 

1 

Answered 

LO_C 

Steering 

1 

1 

Answered 

LO_F 

Diesel  Engines 

VM201_EXAM_7 

1 

1 

Answered 

LO_F 

Diesel  Engines 

VM201_EXAM_8 

1 

1 

EOC  Exam  Scores 

Student_l 

Student_2 

Exam  Score 

Exam  Score 

Passed 

VM101 

87.00% 

100.00% 

Passed 

VM201 

75.00% 

87.00% 
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